mirror of
https://github.com/storytold/vectorcraft.git
synced 2026-11-03 06:30:00 -06:00
Fonts: missing fonts are listed when a document opens, with Find Fonts and a folder search
When a document opens with fonts that aren't available, fonts used
only in its symbols included, VectorCraft lists them in a Missing
Fonts dialog after the missing linked file questions, one document at
a time, with Find Fonts and Close. Find Fonts opens Type › Find Font
on the first missing font of the layers' type, to replace it. After
Find Fonts or Close the dialog doesn't open again for that open; when
another dialog replaces it, it opens again once that dialog closes.
Search Fonts Folder (the Fonts folder File › Package writes next to
the document) and Find in Folder… search a folder and its subfolders
for the fonts' files, and Add Fonts copies the files the user chooses
into a Fonts folder next to the preferences. The desktop app and
vectorcraft-cli read that folder with the system's font folders. Type
› Find Font has a Find in Folder… button too when fonts the document
uses are missing. The web app shows no dialog; a status message says
how many fonts aren't available.
- text.missingFonts lists each missing family and style once,
whatever version the document's type names, and leaves out CSS
generic families such as sans-serif. text.findFontFiles searches a
folder on up to 8 threads and returns at once; text.findFontFiles {}
reports the search, and {stop: true} stops it. A search reads only
what the font scan reads of font files (their table directories and
name, fvar and OS/2 tables) and, on macOS, of suitcase fonts, and
lists a file for a font only when the file makes the font resolve
exactly. It does not enter app packages, the Program Files and
ProgramData folders, the system's folders at the root of a volume,
the Shared and Public folders in Users, a home folder's Library,
AppData, Applications, snap and dot folders, a Library folder's app
data folders or the folders the environment names for apps, and it
skips other hidden folders and does not follow links below the
picked folder. It ends after 60 s by default, after 10,000,000
entries listed, with more than 1,000,000 folders waiting, after
10,000 font files read or 1,000 kept, or once every font has a file;
Stop and Close end it too.
- text.addFontFiles copies only files the last search found. Of
those, it copies only regular .ttf, .otf, .ttc, .otc, .woff and
.woff2 files and suitcase fonts with their resource fork, at most
256 MB a file and 1 GB in all. It never replaces a file: a file with
the same contents is kept, and when another file has that name, the
copy gets a number (Name 2.otf). The fonts are then scanned again,
and type set in them redraws.
- ui.missingFontsDialog and ui.findFontsInFolder open the dialog for
agents; docs/control-protocol.md lists its fields.
- File › Package gives a font file whose name has no font extension
the extension that matches its data (.ttf, .otf or .ttc).
- vectorcraft-text adds set_app_font_dir, which system_font_dirs
reads, WantedFonts, which reads a font file's names as the font scan
does, is_font_file and is_suitcase. #762 split its face reader, which
now takes read budgets: file_face_names_within tries the suitcase
reader first, and reader_face_names_within applies the budgets to
each font. vectorcraft-format adds write_new_with.
Docs: docs/mcp.md, docs/control-protocol.md, ROADMAP.
This commit is contained in:
dotCooCoo
@@ -0,0 +1,17 @@
|
||||
//! Looking through a folder the user picks, and its subfolders, for files of some kind, on separate
|
||||
//! threads: up to 8 walker threads share one stack of folders and list each folder once. They
|
||||
//! follow no links and don't enter other apps' or the system's folders, hidden folders or folders
|
||||
//! whose contents aren't on the disk ([`Rules`]); the time, the entries listed, the folders queued,
|
||||
//! the depth and the files read are capped ([`Limits`]). [`start`] returns at once,
|
||||
//! [`Search::progress`] reports where a search stands and [`Search::stop`] ends it. A search reads
|
||||
//! the files [`Visitor::wants`] accepts by their names and [`Visitor::worth_reading`] by the files
|
||||
//! themselves, and keeps those [`Visitor::items`] gives wanted items for.
|
||||
//! The web has no file system: [`start`] fails there.
|
||||
|
||||
mod rules;
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
mod walk;
|
||||
|
||||
pub use rules::{Rules, plain};
|
||||
pub use walk::{End, Limits, Progress, Search, Visitor, start, threads};
|
||||
@@ -0,0 +1,409 @@
|
||||
//! Where a folder search goes: not into other apps' or the system's folders. They are told by
|
||||
//! their names and places (an app package, a home folder's `Library` or `AppData`, the system
|
||||
//! folders at the root of a volume), never by the name of an app. Names and paths compare in
|
||||
//! lowercase.
|
||||
|
||||
use std::path::{Component, Path, PathBuf};
|
||||
|
||||
/// Extensions of the packages apps and media libraries keep their files in.
|
||||
const PACKAGE_EXTENSIONS: &[&str] = &[
|
||||
"app",
|
||||
"appex",
|
||||
"bundle",
|
||||
"component",
|
||||
"fcpbundle",
|
||||
"framework",
|
||||
"imovielibrary",
|
||||
"kext",
|
||||
"musiclibrary",
|
||||
"photolibrary",
|
||||
"photoslibrary",
|
||||
"plugin",
|
||||
"tvlibrary",
|
||||
"xpc",
|
||||
];
|
||||
|
||||
/// The folders Windows installs programs and keeps their data and the system's in, wherever they
|
||||
/// are.
|
||||
const INSTALL_FOLDERS: &[&str] =
|
||||
&["program files", "program files (x86)", "program files (arm)", "programdata", "$recycle.bin", "system volume information"];
|
||||
|
||||
/// The folders a home folder keeps apps and their data in (besides its hidden ones).
|
||||
const HOME_APP_FOLDERS: &[&str] = &["library", "appdata", "applications", "snap"];
|
||||
|
||||
/// The folders a `Library` folder keeps apps' data in (macOS).
|
||||
const LIBRARY_APP_FOLDERS: &[&str] = &["application support", "containers", "group containers", "caches", "preferences"];
|
||||
|
||||
/// The folders beside the home folders that apps share (macOS `Shared`, Windows `Public`).
|
||||
const SHARED_FOLDERS: &[&str] = &["shared", "public"];
|
||||
|
||||
/// The system's folders at the root of a macOS volume (or a copy of one).
|
||||
const MAC_SYSTEM: &[&str] =
|
||||
&["applications", "bin", "cores", "dev", "etc", "home", "library", "net", "network", "opt", "private", "sbin", "system", "tmp", "usr", "var"];
|
||||
|
||||
/// The system's folders at the root of a Linux or BSD system (or a copy of one).
|
||||
const UNIX_SYSTEM: &[&str] = &[
|
||||
"bin",
|
||||
"boot",
|
||||
"dev",
|
||||
"efi",
|
||||
"etc",
|
||||
"gnu",
|
||||
"lib",
|
||||
"lib32",
|
||||
"lib64",
|
||||
"libx32",
|
||||
"lost+found",
|
||||
"nix",
|
||||
"opt",
|
||||
"proc",
|
||||
"root",
|
||||
"run",
|
||||
"sbin",
|
||||
"snap",
|
||||
"sys",
|
||||
"usr",
|
||||
"var",
|
||||
];
|
||||
|
||||
/// The system's folders at the root of a Windows drive (or a copy of one), besides the install
|
||||
/// folders.
|
||||
const WINDOWS_SYSTEM: &[&str] =
|
||||
&["windows", "windows.old", "recovery", "perflogs", "boot", "config.msi", "msocache", "$windows.~bt", "$windows.~ws", "$winreagent", "$sysreset"];
|
||||
|
||||
/// The most entries of a folder read to tell what kind of folder it is.
|
||||
const MAX_SIBLINGS: usize = 100_000;
|
||||
|
||||
/// The names of the folders that show a folder is the root of a system ([`system_folders`]).
|
||||
const SYSTEM_SIGNS: &[&str] = &["users", "system", "library", "applications", "windows", "program files", "etc", "usr", "bin", "sbin"];
|
||||
|
||||
/// Which of [`SYSTEM_SIGNS`] are among the names of a folder's folders and links.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub(crate) struct Siblings(u16);
|
||||
|
||||
impl Siblings {
|
||||
/// The folders and links named `names` (in any case).
|
||||
#[cfg(test)]
|
||||
pub(crate) fn of(names: &[&str]) -> Self {
|
||||
let mut s = Self::default();
|
||||
names.iter().for_each(|n| s.note(n));
|
||||
s
|
||||
}
|
||||
|
||||
/// Note a folder or link named `name` (in any case).
|
||||
pub(crate) fn note(&mut self, name: &str) {
|
||||
if let Some(i) = SYSTEM_SIGNS.iter().position(|s| s.eq_ignore_ascii_case(name)) {
|
||||
self.0 |= 1 << i;
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a folder or link named `name`, one of [`SYSTEM_SIGNS`], was noted.
|
||||
fn has(self, name: &str) -> bool {
|
||||
SYSTEM_SIGNS.iter().position(|s| *s == name).is_some_and(|i| self.0 & (1 << i) != 0)
|
||||
}
|
||||
}
|
||||
|
||||
/// Why a search may not start in a folder ([`Rules::refusal`]).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum Refusal {
|
||||
/// The folder is in other apps' or the system's folders, or in an app package.
|
||||
Apps,
|
||||
/// The folder is in the Shared or Public folder of a `Users` folder.
|
||||
Shared,
|
||||
}
|
||||
|
||||
/// A folder a walk lists, as the rules see it ([`Rules::listed`]).
|
||||
pub(crate) struct Listed {
|
||||
/// The folder, in lowercase.
|
||||
dir: PathBuf,
|
||||
/// Its name, in lowercase.
|
||||
name: String,
|
||||
/// The system folders in it, when it is the root of a system.
|
||||
system: Option<&'static [&'static str]>,
|
||||
/// It is a home folder.
|
||||
home: bool,
|
||||
}
|
||||
|
||||
/// Where a search may go.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct Rules {
|
||||
/// The home folder (canonical).
|
||||
pub home: Option<PathBuf>,
|
||||
/// Folders searched when picked, whatever holds them (canonical): the temporary folders, and
|
||||
/// tests' folders.
|
||||
pub user_content: Vec<PathBuf>,
|
||||
/// Other apps' and the system's folders the environment names (canonical): never entered, and a
|
||||
/// search of a folder inside one fails.
|
||||
pub refused: Vec<PathBuf>,
|
||||
}
|
||||
|
||||
/// `path` in lowercase and, on Windows, without its `\\?\` prefix ([`unprefixed`]), for comparing
|
||||
/// paths.
|
||||
fn lower(path: &Path) -> PathBuf {
|
||||
let text = path.to_string_lossy();
|
||||
let plain = if cfg!(windows) { unprefixed(&text) } else { None };
|
||||
PathBuf::from(plain.as_deref().unwrap_or(&text).to_lowercase())
|
||||
}
|
||||
|
||||
/// The system's folders a volume's root holds, when its folders (`siblings`) show it is the root of
|
||||
/// an operating system.
|
||||
fn system_folders(siblings: Siblings) -> Option<&'static [&'static str]> {
|
||||
let has = |name: &str| siblings.has(name);
|
||||
if has("users") && (has("system") || has("library") || has("applications")) {
|
||||
Some(MAC_SYSTEM)
|
||||
} else if has("windows") && (has("users") || has("program files")) {
|
||||
Some(WINDOWS_SYSTEM)
|
||||
} else if has("etc") && has("usr") && (has("bin") || has("sbin")) {
|
||||
Some(UNIX_SYSTEM)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// What the names of the folders and links in `dir` show ([`Siblings`]).
|
||||
fn folder_siblings(dir: &Path) -> Siblings {
|
||||
let mut siblings = Siblings::default();
|
||||
for e in std::fs::read_dir(dir).into_iter().flatten().take(MAX_SIBLINGS).flatten() {
|
||||
let name = e.file_name();
|
||||
if let Some(name) = name.to_str()
|
||||
&& e.file_type().is_ok_and(|t| t.is_dir() || t.is_symlink())
|
||||
{
|
||||
siblings.note(name);
|
||||
}
|
||||
}
|
||||
siblings
|
||||
}
|
||||
|
||||
/// The first `n` components of `path`.
|
||||
fn prefix(path: &Path, n: usize) -> PathBuf {
|
||||
path.components().take(n).collect()
|
||||
}
|
||||
|
||||
/// `path` without the `\\?\` prefix that `std::fs::canonicalize` gives paths on Windows: `C:\…`
|
||||
/// for `\\?\C:\…` and `\\server\share\…` for `\\?\UNC\server\share\…`, when every name in it can
|
||||
/// be written without the prefix ([`without_verbatim`]). Other paths, and paths on other systems,
|
||||
/// are returned as they are.
|
||||
pub fn plain(path: PathBuf) -> PathBuf {
|
||||
if !cfg!(windows) {
|
||||
return path;
|
||||
}
|
||||
match path.to_str().and_then(without_verbatim) {
|
||||
Some(p) => PathBuf::from(p),
|
||||
None => path,
|
||||
}
|
||||
}
|
||||
|
||||
/// The Windows path `p` without its `\\?\` prefix, when it has one for a drive or a share: `C:\…`
|
||||
/// for `\\?\C:\…` and `\\server\…` for `\\?\UNC\server\…`.
|
||||
pub(crate) fn unprefixed(p: &str) -> Option<String> {
|
||||
if let Some(unc) = p.strip_prefix(r"\\?\UNC\") {
|
||||
return Some(format!(r"\\{unc}"));
|
||||
}
|
||||
let disk = p.strip_prefix(r"\\?\")?;
|
||||
matches!(disk.as_bytes(), [d, b':', b'\\', ..] if d.is_ascii_alphabetic()).then(|| disk.to_string())
|
||||
}
|
||||
|
||||
/// [`unprefixed`], when each name in the path can be written without the prefix ([`portable`]):
|
||||
/// a share's server and share names, and the names below a drive's root or a share.
|
||||
pub(crate) fn without_verbatim(p: &str) -> Option<String> {
|
||||
let plain = unprefixed(p)?;
|
||||
let (names, at_least) = match plain.strip_prefix(r"\\") {
|
||||
Some(unc) => (unc, 2),
|
||||
None => (plain.get(3..)?, 0),
|
||||
};
|
||||
let mut names: Vec<&str> = if names.is_empty() { vec![] } else { names.split('\\').collect() };
|
||||
// A trailing separator.
|
||||
if names.len() > at_least && names.last() == Some(&"") {
|
||||
names.pop();
|
||||
}
|
||||
let ok = names.len() >= at_least && names.iter().all(|n| portable(n));
|
||||
ok.then_some(plain)
|
||||
}
|
||||
|
||||
/// Whether `name` can be a name in a Windows path without the `\\?\` prefix: not `.` or `..`, not
|
||||
/// ending in a dot or a space, without `<>:"/\|?*` or control characters, and not a device name
|
||||
/// (`CON`, `PRN`, `AUX`, `NUL`, `COM0` to `COM9`, `LPT0` to `LPT9`, these with a superscript digit,
|
||||
/// `CONIN$`, `CONOUT$`), with an extension or not.
|
||||
fn portable(name: &str) -> bool {
|
||||
let stem = name.split('.').next().unwrap_or(name).trim_end().to_ascii_lowercase();
|
||||
let numbered = |device: &str| {
|
||||
stem.strip_prefix(device).is_some_and(|n| {
|
||||
let mut n = n.chars();
|
||||
n.next().is_some_and(|c| c.is_ascii_digit() || "¹²³".contains(c)) && n.next().is_none()
|
||||
})
|
||||
};
|
||||
let device = ["con", "prn", "aux", "nul", "conin$", "conout$"].contains(&stem.as_str()) || numbered("com") || numbered("lpt");
|
||||
!name.is_empty()
|
||||
&& name != "."
|
||||
&& name != ".."
|
||||
&& !name.ends_with(['.', ' '])
|
||||
&& !name.chars().any(|c| c < ' ' || "<>:\"/\\|?*".contains(c))
|
||||
&& !device
|
||||
}
|
||||
|
||||
impl Rules {
|
||||
/// This computer's: the home folder, the temporary folders, and the app and system folders the
|
||||
/// environment names.
|
||||
pub fn system() -> Self {
|
||||
let canonical = |p: PathBuf| p.is_absolute().then(|| std::fs::canonicalize(&p).ok()).flatten();
|
||||
let var = |name: &str| std::env::var_os(name).map(PathBuf::from).and_then(canonical);
|
||||
let home = var(if cfg!(windows) { "USERPROFILE" } else { "HOME" });
|
||||
let user_content = [std::env::temp_dir(), PathBuf::from("/tmp")].into_iter().filter_map(canonical).collect();
|
||||
let names: &[&str] = if cfg!(windows) {
|
||||
&["APPDATA", "LOCALAPPDATA", "ProgramData", "ProgramFiles", "ProgramFiles(x86)", "ProgramW6432", "SystemRoot"]
|
||||
} else {
|
||||
&[
|
||||
"XDG_CONFIG_HOME",
|
||||
"XDG_DATA_HOME",
|
||||
"XDG_CACHE_HOME",
|
||||
"XDG_STATE_HOME",
|
||||
"HOST_XDG_CONFIG_HOME",
|
||||
"HOST_XDG_DATA_HOME",
|
||||
"HOST_XDG_CACHE_HOME",
|
||||
"HOST_XDG_STATE_HOME",
|
||||
]
|
||||
};
|
||||
let refused = names.iter().filter_map(|n| var(n)).collect();
|
||||
Rules { home, user_content, refused }
|
||||
}
|
||||
|
||||
/// Whether `dir` is a home folder: the home folder, a folder in a `Users` or `home` folder, or
|
||||
/// a folder the Flatpak document portal exports (`/run/user/<uid>/doc/<id>/<name>`).
|
||||
fn is_home(&self, dir: &Path) -> bool {
|
||||
if self.home.as_ref().is_some_and(|h| lower(h) == lower(dir)) {
|
||||
return true;
|
||||
}
|
||||
let above = dir.parent().and_then(Path::file_name).map(|n| n.to_string_lossy().to_lowercase());
|
||||
if matches!(above.as_deref(), Some("users" | "home")) {
|
||||
return true;
|
||||
}
|
||||
let parts: Vec<String> = dir.components().map(|c| c.as_os_str().to_string_lossy().to_lowercase()).collect();
|
||||
matches!(parts.as_slice(), [_, run, user, uid, doc, _, _] if run == "run" && user == "user" && is_number(uid) && doc == "doc")
|
||||
}
|
||||
|
||||
/// The folder `dir`, whose folders show `siblings`, as the rules see it when a walk lists it:
|
||||
/// told once for all the folders in it.
|
||||
pub(crate) fn listed(&self, dir: &Path, siblings: Siblings) -> Listed {
|
||||
Listed {
|
||||
dir: lower(dir),
|
||||
name: dir.file_name().map(|n| n.to_string_lossy().to_lowercase()).unwrap_or_default(),
|
||||
system: system_folders(siblings),
|
||||
home: self.is_home(dir),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a search never enters the folder `name` in the listed folder `parent`: an app's
|
||||
/// package, a Windows install folder, a system folder at the root of a volume, the folders apps
|
||||
/// share beside the home folders, a `Library` folder's app data, or a home folder's hidden
|
||||
/// folders and app folders.
|
||||
pub(crate) fn never_in(&self, parent: &Listed, name: &str) -> bool {
|
||||
let name = name.to_lowercase();
|
||||
let name = name.as_str();
|
||||
if Path::new(name).extension().and_then(|e| e.to_str()).is_some_and(|e| PACKAGE_EXTENSIONS.contains(&e)) || INSTALL_FOLDERS.contains(&name) {
|
||||
return true;
|
||||
}
|
||||
if parent.system.is_some_and(|system| system.contains(&name)) {
|
||||
return true;
|
||||
}
|
||||
match parent.name.as_str() {
|
||||
"users" if SHARED_FOLDERS.contains(&name) => return true,
|
||||
"library" if LIBRARY_APP_FOLDERS.contains(&name) => return true,
|
||||
_ => {}
|
||||
}
|
||||
parent.home && (name.starts_with('.') || HOME_APP_FOLDERS.contains(&name))
|
||||
}
|
||||
|
||||
/// [`Self::never_in`] the folder `parent`, whose folders show `siblings`.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn never(&self, parent: &Path, siblings: Siblings, name: &str) -> bool {
|
||||
self.never_in(&self.listed(parent, siblings), name)
|
||||
}
|
||||
|
||||
/// The home folder, when the listed folder `parent` is the root of a system whose rules skip
|
||||
/// the folder the home folder is in (`/usr/home/me` on FreeBSD, `/var/home/me`): a walk of the
|
||||
/// root reaches it this way.
|
||||
pub(crate) fn home_below(&self, parent: &Listed) -> Option<PathBuf> {
|
||||
let home = self.home.as_ref()?;
|
||||
let system = parent.system?;
|
||||
let below = lower(home).strip_prefix(&parent.dir).ok()?.components().next()?.as_os_str().to_string_lossy().into_owned();
|
||||
system.contains(&below.as_str()).then(|| home.clone())
|
||||
}
|
||||
|
||||
/// Whether `path` is one of the refused folders.
|
||||
pub(crate) fn is_refused(&self, path: &Path) -> bool {
|
||||
let path = lower(path);
|
||||
self.refused.iter().any(|r| lower(r) == path)
|
||||
}
|
||||
|
||||
/// The folders that hold user content whatever holds them, as far as they hold `folder`: the
|
||||
/// home folder, [`Self::user_content`], the cloud storage folders in the home folder's
|
||||
/// `Library` (an app's own iCloud `Documents` folder too), removable media and a user's runtime
|
||||
/// folder on Linux (network shares, the document portal).
|
||||
fn bases(&self, folder: &Path) -> Vec<PathBuf> {
|
||||
let mut v = self.user_content.clone();
|
||||
let lowered = lower(folder);
|
||||
if let Some(home) = &self.home {
|
||||
v.push(home.clone());
|
||||
let library = home.join("Library");
|
||||
v.push(library.join("CloudStorage"));
|
||||
let mobile = library.join("Mobile Documents");
|
||||
v.push(mobile.join("com~apple~CloudDocs"));
|
||||
let depth = mobile.components().count();
|
||||
if lowered.starts_with(lower(&mobile)) && lowered.components().nth(depth + 1).is_some_and(|c| c.as_os_str() == "documents") {
|
||||
v.push(prefix(folder, depth + 2));
|
||||
}
|
||||
}
|
||||
v.push(PathBuf::from("/run/media"));
|
||||
let parts: Vec<String> = folder.components().take(4).map(|c| c.as_os_str().to_string_lossy().to_lowercase()).collect();
|
||||
if matches!(parts.as_slice(), [_, run, user, uid] if run == "run" && user == "user" && is_number(uid)) {
|
||||
v.push(prefix(folder, 4));
|
||||
}
|
||||
v
|
||||
}
|
||||
|
||||
/// Why a search may not start in `folder` (canonical), or `None` when it may: the deepest base
|
||||
/// ([`Self::bases`]) or refused folder that holds it is a refused one, or a folder below that
|
||||
/// base (or below the root, without one) is one a walk doesn't enter.
|
||||
pub(crate) fn refusal(&self, folder: &Path) -> Option<Refusal> {
|
||||
self.refusal_with(folder, folder_siblings)
|
||||
}
|
||||
|
||||
/// [`Self::refusal`] without listing a folder: the root of a system, told by the folders it
|
||||
/// holds, isn't recognized.
|
||||
pub(crate) fn refusal_unlisted(&self, folder: &Path) -> Option<Refusal> {
|
||||
self.refusal_with(folder, |_| Siblings::default())
|
||||
}
|
||||
|
||||
/// [`Self::refusal`], with `siblings` telling what a folder's folders show.
|
||||
fn refusal_with(&self, folder: &Path, siblings: impl Fn(&Path) -> Siblings) -> Option<Refusal> {
|
||||
let lowered = lower(folder);
|
||||
let held = |p: &PathBuf| lowered.starts_with(lower(p)).then(|| p.components().count());
|
||||
let base = self.bases(folder).iter().filter_map(held).max();
|
||||
let refused = self.refused.iter().filter_map(held).max();
|
||||
let start = match (base, refused) {
|
||||
(_, Some(r)) if base.is_none_or(|b| r >= b) => return Some(Refusal::Apps),
|
||||
(Some(b), _) => b,
|
||||
// From the root of the folder's volume.
|
||||
(None, _) => folder.components().take_while(|c| matches!(c, Component::Prefix(_) | Component::RootDir)).count(),
|
||||
};
|
||||
let mut parent = prefix(folder, start);
|
||||
for c in folder.components().skip(start) {
|
||||
let child = parent.join(c);
|
||||
let listed = self.listed(&parent, siblings(&parent));
|
||||
let name = c.as_os_str().to_string_lossy().to_lowercase();
|
||||
if listed.name == "users" && SHARED_FOLDERS.contains(&name.as_str()) {
|
||||
return Some(Refusal::Shared);
|
||||
}
|
||||
if self.never_in(&listed, &name) || self.is_refused(&child) {
|
||||
return Some(Refusal::Apps);
|
||||
}
|
||||
parent = child;
|
||||
}
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `s` is a number (a user id).
|
||||
fn is_number(s: &str) -> bool {
|
||||
!s.is_empty() && s.bytes().all(|b| b.is_ascii_digit())
|
||||
}
|
||||
@@ -0,0 +1,545 @@
|
||||
//! The folder search: where it goes and where it doesn't, its limits, its threads.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::{Arc, Mutex, mpsc};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use super::rules::{Siblings, unprefixed, without_verbatim};
|
||||
use super::walk::{Pool, start_in};
|
||||
use super::*;
|
||||
|
||||
/// The walker threads of these tests' searches, apart from the searches of other tests.
|
||||
static TESTS: Pool = Pool::new(32);
|
||||
|
||||
/// [`start`] on the threads of [`TESTS`].
|
||||
fn begin(folder: PathBuf, rules: Rules, limits: Limits, threads: usize, items: usize, visitor: Arc<dyn Visitor>) -> Search {
|
||||
start_in(&TESTS, folder, Some(rules), limits, threads, items, visitor).unwrap()
|
||||
}
|
||||
|
||||
/// Reads `.ttf` files; each is the item its name is in the list.
|
||||
struct Names(&'static [&'static str]);
|
||||
|
||||
impl Visitor for Names {
|
||||
fn wants(&self, name: &str) -> bool {
|
||||
name.ends_with(".ttf")
|
||||
}
|
||||
fn items(&self, path: &Path) -> Vec<usize> {
|
||||
let name = path.file_name().and_then(|n| n.to_str()).unwrap_or("");
|
||||
self.0.iter().position(|n| *n == name).into_iter().collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// Reads every file, and waits on reading one until its channel sends or closes.
|
||||
struct Blocked(Mutex<mpsc::Receiver<()>>);
|
||||
|
||||
impl Visitor for Blocked {
|
||||
fn wants(&self, _: &str) -> bool {
|
||||
true
|
||||
}
|
||||
fn items(&self, _: &Path) -> Vec<usize> {
|
||||
let _ = self.0.lock().unwrap().recv();
|
||||
vec![]
|
||||
}
|
||||
}
|
||||
|
||||
/// A fresh folder (canonical, as a search reports it) holding `paths`: files, and folders where
|
||||
/// they end in `/`.
|
||||
fn tree(tag: &str, paths: &[&str]) -> PathBuf {
|
||||
let root = vectorcraft_testkit::temp_dir(&format!("findfiles-{tag}"));
|
||||
let _ = std::fs::remove_dir_all(&root);
|
||||
std::fs::create_dir_all(&root).unwrap();
|
||||
for p in paths {
|
||||
let path = root.join(p);
|
||||
if p.ends_with('/') {
|
||||
std::fs::create_dir_all(&path).unwrap();
|
||||
} else {
|
||||
std::fs::create_dir_all(path.parent().unwrap()).unwrap();
|
||||
std::fs::write(&path, b"font").unwrap();
|
||||
}
|
||||
}
|
||||
plain(std::fs::canonicalize(&root).unwrap())
|
||||
}
|
||||
|
||||
/// The rules of a search of `root`: its `Users/me` is the home folder, `refused` an app's folder.
|
||||
fn rules(root: &Path) -> Rules {
|
||||
Rules { home: Some(root.join("Users/me")), user_content: vec![root.to_path_buf()], refused: vec![root.join("refused")] }
|
||||
}
|
||||
|
||||
/// The search's progress once it ended (or after 10 seconds).
|
||||
fn wait(s: &Search) -> Progress {
|
||||
let until = Instant::now() + Duration::from_secs(10);
|
||||
loop {
|
||||
let p = s.progress();
|
||||
if p.end.is_some() || Instant::now() > until {
|
||||
return p;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
}
|
||||
|
||||
/// Search `from` in `root` for the files named `names`, to the end.
|
||||
fn search(root: &Path, from: &str, limits: Limits, threads: usize, names: &'static [&'static str]) -> Progress {
|
||||
let s = begin(root.join(from), rules(root), limits, threads, names.len(), Arc::new(Names(names)));
|
||||
wait(&s)
|
||||
}
|
||||
|
||||
/// The files a search found, relative to `root`, sorted.
|
||||
fn found(root: &Path, p: &Progress) -> Vec<String> {
|
||||
let mut v: Vec<String> = p.hits.iter().map(|(f, _)| f.strip_prefix(root).unwrap().to_string_lossy().replace('\\', "/")).collect();
|
||||
v.sort();
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_walk_stays_out_of_other_apps_and_the_systems_folders() {
|
||||
let root = tree(
|
||||
"roles",
|
||||
&[
|
||||
"a/b/c/one.ttf",
|
||||
"Users/me/Documents/two.ttf",
|
||||
"Disk/Fonts/four.ttf",
|
||||
"Disk/Users/you/Documents/four.ttf",
|
||||
// Never entered: each of these folders counts as skipped once.
|
||||
"Program Files/x/three.ttf",
|
||||
"Program Files (Arm)/x/three.ttf",
|
||||
"Users/me/Library/Fonts/three.ttf",
|
||||
"Users/me/.cache/three.ttf",
|
||||
"Users/me/AppData/Roaming/three.ttf",
|
||||
"Users/you/snap/three.ttf",
|
||||
"Users/Shared/x/three.ttf",
|
||||
"Pictures/P.photoslibrary/three.ttf",
|
||||
"Tools/Foo.app/Contents/three.ttf",
|
||||
"Copy/Library/Application Support/x/three.ttf",
|
||||
".hidden/three.ttf",
|
||||
"refused/three.ttf",
|
||||
"Disk/System/Library/Fonts/three.ttf",
|
||||
"Disk/Library/Fonts/three.ttf",
|
||||
"Disk/Applications/three.ttf",
|
||||
"Disk/Users/you/Library/three.ttf",
|
||||
],
|
||||
);
|
||||
let p = search(&root, "", Limits::default(), 3, &["one.ttf", "two.ttf", "three.ttf", "four.ttf"]);
|
||||
assert_eq!(p.end, Some(End::Done));
|
||||
assert_eq!(found(&root, &p), ["Disk/Fonts/four.ttf", "Disk/Users/you/Documents/four.ttf", "Users/me/Documents/two.ttf", "a/b/c/one.ttf"]);
|
||||
assert_eq!(p.skipped, 16);
|
||||
assert_eq!((p.read, p.unreadable), (4, 0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_picked_folder_inside_other_apps_folders_is_refused() {
|
||||
let root = tree(
|
||||
"picks",
|
||||
&[
|
||||
"Users/me/Library/Fonts/",
|
||||
"Users/me/Library/CloudStorage/Box/F/",
|
||||
"Users/me/Library/Mobile Documents/x~y/Documents/",
|
||||
"Users/me/Library/Mobile Documents/x~y/Other/",
|
||||
"Users/me/.config/fonts/",
|
||||
"Users/Shared/x/",
|
||||
"Disk/Users/",
|
||||
"Disk/System/",
|
||||
"Disk/Library/Fonts/",
|
||||
"Tools/Foo.app/Contents/",
|
||||
"refused/sub/",
|
||||
"Program Files/x/",
|
||||
"Program Files (Arm)/x/",
|
||||
"Users/Public/x/",
|
||||
"a/.b/",
|
||||
"file.ttf",
|
||||
],
|
||||
);
|
||||
for picked in [
|
||||
"Users/me/Library/Fonts",
|
||||
"Users/me/Library/Mobile Documents/x~y/Other",
|
||||
"Users/me/.config/fonts",
|
||||
"Disk/Library/Fonts",
|
||||
"Tools/Foo.app/Contents",
|
||||
"refused",
|
||||
"refused/sub",
|
||||
"Program Files/x",
|
||||
"Program Files (Arm)/x",
|
||||
] {
|
||||
let p = search(&root, picked, Limits::default(), 1, &[]);
|
||||
assert_eq!(p.state(), "failed", "{picked}");
|
||||
assert!(p.error().is_some_and(|e| e.contains("doesn't search other apps' or the system's folders")), "{picked}: {p:?}");
|
||||
}
|
||||
for picked in ["Users/Shared/x", "Users/Public/x"] {
|
||||
let p = search(&root, picked, Limits::default(), 1, &[]);
|
||||
assert!(p.error().is_some_and(|e| e.contains("doesn't search the Shared or Public folder in Users")), "{picked}: {p:?}");
|
||||
}
|
||||
// Cloud folders, a hidden folder of the user's and another system's root are searched when
|
||||
// picked.
|
||||
for picked in ["Users/me/Library/CloudStorage/Box/F", "Users/me/Library/Mobile Documents/x~y/Documents", "a/.b", "Disk", "Disk/Users", ""] {
|
||||
let p = search(&root, picked, Limits::default(), 1, &[]);
|
||||
assert_eq!((p.state(), p.error()), ("done", None), "{picked}");
|
||||
}
|
||||
let p = search(&root, "file.ttf", Limits::default(), 1, &[]);
|
||||
assert!(p.error().is_some_and(|e| e.ends_with("is not a folder")), "{p:?}");
|
||||
let p = search(&root, "missing", Limits::default(), 1, &[]);
|
||||
assert!(p.error().is_some_and(|e| e.starts_with(&root.join("missing").display().to_string())), "{p:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_search_ends_at_its_limits_and_when_stopped() {
|
||||
let root = tree("limits", &["one.ttf", "two.ttf", "a/b/c/three.ttf", "x/y.txt"]);
|
||||
let names = &["one.ttf", "two.ttf", "three.ttf", "nowhere.ttf"];
|
||||
// The picked folder and the folders in it are listed, not the ones below them.
|
||||
let p = search(&root, "", Limits { depth: 1, ..Limits::default() }, 2, names);
|
||||
assert_eq!(p.end, Some(End::Done));
|
||||
assert_eq!(found(&root, &p), ["one.ttf", "two.ttf"]);
|
||||
assert_eq!((p.folders, p.skipped), (3, 1));
|
||||
let p = search(&root, "", Limits { entries: 3, ..Limits::default() }, 1, names);
|
||||
assert_eq!((p.state(), p.stopped()), ("stopped", Some("limit")));
|
||||
let p = search(&root, "", Limits { files: 1, ..Limits::default() }, 1, names);
|
||||
assert_eq!((p.stopped(), p.read), (Some("limit"), 1));
|
||||
let p = search(&root, "", Limits { hits: 1, ..Limits::default() }, 1, names);
|
||||
assert_eq!((p.stopped(), p.hits.len()), (Some("limit"), 1));
|
||||
// Each item keeps at most `per_item` files.
|
||||
let root = tree("per-item", &["a/one.ttf", "b/one.ttf", "c/one.ttf"]);
|
||||
let p = search(&root, "", Limits { per_item: 2, ..Limits::default() }, 1, &["one.ttf", "nowhere.ttf"]);
|
||||
assert_eq!((p.end, p.hits.len(), p.read), (Some(End::Done), 2, 3));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_search_answers_at_its_deadline_while_a_walker_is_blocked() {
|
||||
let root = tree("blocked", &["f.ttf"]);
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let s = begin(root.clone(), rules(&root), Limits { seconds: 0.2, ..Limits::default() }, 1, 1, Arc::new(Blocked(Mutex::new(rx))));
|
||||
std::thread::sleep(Duration::from_millis(300));
|
||||
let p = s.progress();
|
||||
assert_eq!((p.state(), p.stopped()), ("stopped", Some("time")), "{p:?}");
|
||||
assert!(p.seconds >= 0.2 && p.seconds < 5.0, "{p:?}");
|
||||
let (tx2, rx2) = mpsc::channel();
|
||||
let s2 = begin(root.clone(), rules(&root), Limits::default(), 1, 1, Arc::new(Blocked(Mutex::new(rx2))));
|
||||
let until = Instant::now() + Duration::from_secs(10);
|
||||
while s2.progress().read == 0 && Instant::now() < until {
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
assert_eq!(s2.progress().state(), "searching");
|
||||
s2.stop();
|
||||
assert_eq!(s2.progress().stopped(), Some("stop"));
|
||||
drop((tx, tx2));
|
||||
}
|
||||
|
||||
/// Every file is item 0; reading the second file waits until the channel closes.
|
||||
struct SecondBlocks(Mutex<(usize, mpsc::Receiver<()>)>);
|
||||
|
||||
impl Visitor for SecondBlocks {
|
||||
fn wants(&self, _: &str) -> bool {
|
||||
true
|
||||
}
|
||||
fn items(&self, _: &Path) -> Vec<usize> {
|
||||
let mut read = self.0.lock().unwrap();
|
||||
read.0 += 1;
|
||||
if read.0 == 2 {
|
||||
let _ = read.1.recv();
|
||||
}
|
||||
vec![0]
|
||||
}
|
||||
}
|
||||
|
||||
/// A file is kept as soon as it is read: progress shows it while the rest of its folder is read,
|
||||
/// and a search that stops or reaches a limit keeps it.
|
||||
#[test]
|
||||
fn files_found_are_kept_as_they_are_read() {
|
||||
let root = tree("kept", &["a.ttf", "b.ttf", "c.ttf"]);
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let s = begin(root.clone(), rules(&root), Limits::default(), 1, 2, Arc::new(SecondBlocks(Mutex::new((0, rx)))));
|
||||
let until = Instant::now() + Duration::from_secs(10);
|
||||
while s.progress().read < 2 && Instant::now() < until {
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
let p = s.progress();
|
||||
assert_eq!((p.state(), p.hits.len()), ("searching", 1), "{p:?}");
|
||||
s.stop();
|
||||
drop(tx);
|
||||
let p = wait(&s);
|
||||
assert_eq!(p.stopped(), Some("stop"));
|
||||
assert!(!p.hits.is_empty(), "{p:?}");
|
||||
let p = search(&root, "", Limits { files: 2, ..Limits::default() }, 1, &["a.ttf", "b.ttf", "c.ttf", "nowhere.ttf"]);
|
||||
assert_eq!((p.stopped(), p.hits.len()), (Some("limit"), 2), "{p:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_search_ends_once_every_item_has_a_file() {
|
||||
let root = tree("found", &["a/one.ttf", "b/two.ttf", "c/d/e/f/g/other.ttf"]);
|
||||
let p = search(&root, "", Limits::default(), 2, &["one.ttf", "two.ttf"]);
|
||||
assert_eq!((p.end.clone(), found(&root, &p)), (Some(End::Found), vec!["a/one.ttf".to_string(), "b/two.ttf".into()]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn one_walker_and_eight_find_the_same_files_and_every_search_ends() {
|
||||
let mut paths = vec![];
|
||||
for a in 0..6 {
|
||||
for b in 0..4 {
|
||||
paths.push(format!("d{a}/e{b}/one.ttf"));
|
||||
paths.push(format!("d{a}/e{b}/f/two.ttf"));
|
||||
paths.push(format!("d{a}/e{b}/f/g/skip.txt"));
|
||||
}
|
||||
}
|
||||
let paths: Vec<&str> = paths.iter().map(String::as_str).collect();
|
||||
let root = tree("threads", &paths);
|
||||
let limits = Limits { per_item: 100, ..Limits::default() };
|
||||
let one = search(&root, "", limits, 1, &["one.ttf", "two.ttf", "nowhere.ttf"]);
|
||||
assert_eq!((one.end.clone(), one.hits.len(), one.folders), (Some(End::Done), 48, 1 + 6 + 24 + 24 + 24));
|
||||
for round in 0..25 {
|
||||
for threads in 1..=8 {
|
||||
let p = search(&root, "", limits, threads, &["one.ttf", "two.ttf", "nowhere.ttf"]);
|
||||
assert_eq!(p.end, Some(End::Done), "round {round}, {threads} threads");
|
||||
assert_eq!(found(&root, &p), found(&root, &one), "round {round}, {threads} threads");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Fails on `boom.ttf` with a panic: a bug in reading one file.
|
||||
struct Boom;
|
||||
|
||||
impl Visitor for Boom {
|
||||
fn wants(&self, name: &str) -> bool {
|
||||
name.ends_with(".ttf")
|
||||
}
|
||||
fn items(&self, path: &Path) -> Vec<usize> {
|
||||
match path.file_name().and_then(|n| n.to_str()) {
|
||||
Some("boom.ttf") => panic!("a bug"),
|
||||
Some("ok.ttf") => vec![0],
|
||||
_ => vec![1],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_visitor_that_panics_on_one_file_does_not_end_the_search() {
|
||||
// The file beside the one that trips the bug is found too.
|
||||
let root = tree("boom", &["a/boom.ttf", "a/ok.ttf", "b/c/fine.ttf"]);
|
||||
let s = begin(root.clone(), rules(&root), Limits::default(), 2, 3, Arc::new(Boom));
|
||||
let p = wait(&s);
|
||||
assert_eq!((p.end.clone(), found(&root, &p), p.unreadable), (Some(End::Done), vec!["a/ok.ttf".to_string(), "b/c/fine.ttf".into()], 1));
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn links_are_not_followed_and_unreadable_folders_count() {
|
||||
use std::os::unix::fs::{PermissionsExt, symlink};
|
||||
let root = tree("links", &["outside/x.ttf", "inner/locked/y.ttf"]);
|
||||
let inner = root.join("inner");
|
||||
symlink(root.join("outside"), inner.join("folder")).unwrap();
|
||||
symlink(&inner, inner.join("loop")).unwrap();
|
||||
symlink(root.join("outside/x.ttf"), inner.join("x.ttf")).unwrap();
|
||||
let locked = inner.join("locked");
|
||||
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o000)).unwrap();
|
||||
// Where permissions don't hold (root), the folder reads as any other.
|
||||
let enforced = std::fs::read_dir(&locked).is_err();
|
||||
let p = search(&root, "inner", Limits::default(), 2, &["x.ttf", "y.ttf"]);
|
||||
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o755)).unwrap();
|
||||
assert_eq!(p.end, Some(End::Done));
|
||||
if enforced {
|
||||
assert!(p.hits.is_empty() && p.unreadable == 1, "{p:?}");
|
||||
} else {
|
||||
assert_eq!(found(&root, &p), ["inner/locked/y.ttf"]);
|
||||
}
|
||||
// The linked file isn't a file of the folder; y.ttf is one where the locked folder reads.
|
||||
assert_eq!(p.files, u64::from(!enforced), "{p:?}");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn a_picked_folder_that_cant_be_listed_fails_the_search() {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let root = tree("unlistable", &["locked/y.ttf"]);
|
||||
let locked = root.join("locked");
|
||||
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o000)).unwrap();
|
||||
// Where permissions don't hold (root), the folder reads as any other.
|
||||
let enforced = std::fs::read_dir(&locked).is_err();
|
||||
let p = search(&root, "locked", Limits::default(), 2, &["y.ttf"]);
|
||||
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o755)).unwrap();
|
||||
if enforced {
|
||||
assert!(matches!(&p.end, Some(End::Failed(e)) if e.contains("locked")), "{p:?}");
|
||||
} else {
|
||||
assert_eq!(found(&root, &p), ["locked/y.ttf"]);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_roots_and_homes_are_told_by_what_they_hold() {
|
||||
let r = Rules { home: Some(PathBuf::from("/home/me")), ..Rules::default() };
|
||||
let never = |parent: &str, siblings: Siblings, name: &str| r.never(Path::new(parent), siblings, name);
|
||||
let none = Siblings::default();
|
||||
let mac = Siblings::of(&["Users", "Library", "System", "Applications", "Volumes", "opt"]);
|
||||
for name in ["Library", "System", "Applications", "opt", "private"] {
|
||||
assert!(never("/", mac, name), "{name}");
|
||||
}
|
||||
for name in ["Users", "Volumes", "Fonts"] {
|
||||
assert!(!never("/", mac, name), "{name}");
|
||||
}
|
||||
let unix = Siblings::of(&["usr", "etc", "bin", "home", "nix", "srv", "data", "mnt"]);
|
||||
for name in ["usr", "etc", "bin", "nix", "var", "proc"] {
|
||||
assert!(never("/", unix, name), "{name}");
|
||||
}
|
||||
for name in ["home", "srv", "data", "mnt", "media"] {
|
||||
assert!(!never("/", unix, name), "{name}");
|
||||
}
|
||||
let windows = Siblings::of(&["Windows", "Users", "Program Files", "Data"]);
|
||||
for name in ["Windows", "Program Files", "Recovery"] {
|
||||
assert!(never("C:\\", windows, name), "{name}");
|
||||
}
|
||||
for name in ["Data", "Users"] {
|
||||
assert!(!never("C:\\", windows, name), "{name}");
|
||||
}
|
||||
// A project that holds a bin and an etc folder isn't the root of a system.
|
||||
let project = Siblings::of(&["bin", "etc", "src"]);
|
||||
assert!(!never("/work/p", project, "bin") && !never("/work/p", project, "etc"));
|
||||
// Home folders: the home folder, and every folder in a Users or home folder.
|
||||
for name in ["Library", ".config", "snap", "AppData", "Applications"] {
|
||||
assert!(never("/home/me", none, name), "{name}");
|
||||
assert!(never("/Users/you", none, name), "{name}");
|
||||
}
|
||||
assert!(!never("/home/me", none, "Documents") && !never("/home/me/Documents", none, "Library"));
|
||||
assert!(never("/run/user/1000/doc/a1b2/me", none, "snap"), "a folder the document portal exports");
|
||||
assert!(never("/Users", none, "Shared") && never("/Users", none, "Public"));
|
||||
assert!(never("/Volumes/Copy/Library", none, "Application Support") && !never("/Volumes/Copy/Library", none, "Fonts"));
|
||||
assert!(never("/x", none, "Foo.app") && never("/x", none, "Program Files (x86)") && !never("/x", none, "Fonts.d"));
|
||||
assert!(never("/x", none, "Program Files (Arm)"));
|
||||
// A system root's walk reaches a home folder below a folder it skips (FreeBSD's /usr/home).
|
||||
let bsd = Rules { home: Some(PathBuf::from("/usr/home/me")), ..Rules::default() };
|
||||
let root = Path::new("/");
|
||||
assert_eq!(bsd.home_below(&bsd.listed(root, unix)), Some(PathBuf::from("/usr/home/me")));
|
||||
assert_eq!(r.home_below(&r.listed(root, unix)), None);
|
||||
assert_eq!(bsd.home_below(&bsd.listed(root, project)), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_search_starts_with_the_threads_left_and_fails_when_none_is() {
|
||||
static POOL: Pool = Pool::new(3);
|
||||
let root = tree("pool", &["f.ttf"]);
|
||||
let blocked = || {
|
||||
let (tx, rx) = mpsc::channel();
|
||||
(tx, Arc::new(Blocked(Mutex::new(rx))) as Arc<dyn Visitor>)
|
||||
};
|
||||
let (tx_a, a) = blocked();
|
||||
let (tx_b, b) = blocked();
|
||||
let a = start_in(&POOL, root.clone(), Some(rules(&root)), Limits::default(), 2, 1, a).unwrap();
|
||||
let b = start_in(&POOL, root.clone(), Some(rules(&root)), Limits::default(), 4, 1, b).unwrap();
|
||||
assert_eq!(POOL.live(), 3);
|
||||
let (_, c) = blocked();
|
||||
let c = start_in(&POOL, root.clone(), Some(rules(&root)), Limits::default(), 1, 1, c);
|
||||
assert_eq!(c.err().as_deref(), Some("the search couldn't start: try again in a moment"));
|
||||
drop((tx_a, tx_b));
|
||||
assert_eq!((wait(&a).end, wait(&b).end), (Some(End::Done), Some(End::Done)));
|
||||
let until = Instant::now() + Duration::from_secs(10);
|
||||
while POOL.live() > 0 && Instant::now() < until {
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
}
|
||||
assert_eq!(POOL.live(), 0, "every walker thread ended");
|
||||
let s = start_in(&POOL, root.clone(), Some(rules(&root)), Limits::default(), 8, 1, Arc::new(Names(&[]))).unwrap();
|
||||
assert_eq!(wait(&s).end, Some(End::Done));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_search_takes_the_cpus_the_renderer_leaves_from_two_to_eight() {
|
||||
use super::walk::threads_for;
|
||||
assert_eq!([threads_for(64, 4), threads_for(12, 4), threads_for(4, 4), threads_for(1, 0)], [8, 7, 2, 2]);
|
||||
assert!((2..=8).contains(&threads()));
|
||||
}
|
||||
|
||||
/// A folder holding more folders than a search queues ends the search while it is being listed.
|
||||
#[test]
|
||||
fn a_folder_with_more_folders_than_the_search_queues_ends_it_as_it_is_listed() {
|
||||
// Folders and files made in turns: whatever order a file system lists them in, about half of a
|
||||
// folder's first entries are folders.
|
||||
let names: Vec<String> = (0..2048).flat_map(|i| [format!("d{i}/"), format!("f{i}.txt")]).collect();
|
||||
let names: Vec<&str> = names.iter().map(String::as_str).collect();
|
||||
let root = tree("many-folders", &names);
|
||||
let p = search(&root, "", Limits { folders: 10, ..Limits::default() }, 1, &[]);
|
||||
assert_eq!(p.end, Some(End::Limit));
|
||||
assert!(p.files < 2048, "the folder was listed to its end: {p:?}");
|
||||
}
|
||||
|
||||
/// Wants every file, counting the files it is asked about.
|
||||
struct Counted(AtomicUsize);
|
||||
|
||||
impl Visitor for Counted {
|
||||
fn wants(&self, _: &str) -> bool {
|
||||
self.0.fetch_add(1, Ordering::Relaxed);
|
||||
true
|
||||
}
|
||||
fn items(&self, _: &Path) -> Vec<usize> {
|
||||
vec![]
|
||||
}
|
||||
}
|
||||
|
||||
/// A folder's files past those the search reads aren't kept to be read.
|
||||
#[test]
|
||||
fn files_past_those_the_search_reads_are_not_kept() {
|
||||
let names: Vec<String> = (0..50).map(|i| format!("f{i}.ttf")).collect();
|
||||
let names: Vec<&str> = names.iter().map(String::as_str).collect();
|
||||
let root = tree("many-files", &names);
|
||||
let counted = Arc::new(Counted(AtomicUsize::new(0)));
|
||||
let p = wait(&begin(root.clone(), rules(&root), Limits { files: 3, ..Limits::default() }, 1, 1, counted.clone()));
|
||||
assert_eq!((p.stopped(), p.read, p.files), (Some("limit"), 3, 50), "{p:?}");
|
||||
assert_eq!(counted.0.load(Ordering::Relaxed), 4, "one more than the files read");
|
||||
}
|
||||
|
||||
/// Wants `.ttf` files by their names, turns down those whose names start with `skip` by the file,
|
||||
/// and finds `font.ttf`.
|
||||
struct Picky;
|
||||
|
||||
impl Visitor for Picky {
|
||||
fn wants(&self, name: &str) -> bool {
|
||||
name.ends_with(".ttf")
|
||||
}
|
||||
fn worth_reading(&self, path: &Path) -> bool {
|
||||
!path.file_name().and_then(|n| n.to_str()).is_some_and(|n| n.starts_with("skip"))
|
||||
}
|
||||
fn items(&self, path: &Path) -> Vec<usize> {
|
||||
if path.ends_with("font.ttf") { vec![0] } else { vec![] }
|
||||
}
|
||||
}
|
||||
|
||||
/// Files a visitor turns down by the file itself aren't read and don't count among the files
|
||||
/// read: with room to read one file, the search gets past them to the font in the folder below.
|
||||
#[test]
|
||||
fn files_turned_down_by_the_file_are_not_read() {
|
||||
let root = tree("turned-down", &["skip1.ttf", "skip2.ttf", "skip3.ttf", "sub/font.ttf"]);
|
||||
let p = wait(&begin(root.clone(), rules(&root), Limits { files: 1, ..Limits::default() }, 1, 1, Arc::new(Picky)));
|
||||
assert_eq!((p.end.clone(), p.read, found(&root, &p)), (Some(End::Found), 1, vec!["sub/font.ttf".to_string()]), "{p:?}");
|
||||
}
|
||||
|
||||
/// Windows paths lose the `\\?\` prefix `canonicalize` gives them where they read the same
|
||||
/// without it. Paths compare the same with the prefix and without it.
|
||||
#[test]
|
||||
fn windows_paths_lose_the_verbatim_prefix_where_they_read_the_same() {
|
||||
for (verbatim, without) in [
|
||||
(r"\\?\C:\Users\me\Fonts", r"C:\Users\me\Fonts"),
|
||||
(r"\\?\d:\", r"d:\"),
|
||||
(r"\\?\UNC\server\share\a b\c.otf", r"\\server\share\a b\c.otf"),
|
||||
(r"\\?\UNC\server\share", r"\\server\share"),
|
||||
(r"\\?\UNC\server\share\", r"\\server\share\"),
|
||||
] {
|
||||
assert_eq!(without_verbatim(verbatim).as_deref(), Some(without), "{verbatim}");
|
||||
}
|
||||
// For comparing, any name goes.
|
||||
assert_eq!(unprefixed(r"\\?\C:\a.\b").as_deref(), Some(r"C:\a.\b"));
|
||||
assert_eq!(unprefixed(r"\\?\UNC\server").as_deref(), Some(r"\\server"));
|
||||
assert_eq!((unprefixed(r"\\?\C:"), unprefixed(r"C:\a"), unprefixed("/home/me")), (None, None, None));
|
||||
for kept in [
|
||||
r"\\?\C:",
|
||||
r"\\?\C:\a.\b",
|
||||
r"\\?\C:\a \b",
|
||||
r"\\?\C:\CON\b",
|
||||
r"\\?\C:\a\nul.txt",
|
||||
r"\\?\C:\a\COM¹",
|
||||
r"\\?\C:\a/b",
|
||||
r"\\?\C:\a\..\b",
|
||||
r"\\?\C:\a\\b",
|
||||
r"\\?\Volume{00000000-0000-0000-0000-000000000000}\a",
|
||||
r"\\?\UNC\server",
|
||||
r"C:\a",
|
||||
"/home/me",
|
||||
] {
|
||||
assert_eq!(without_verbatim(kept), None, "{kept}");
|
||||
}
|
||||
if !cfg!(windows) {
|
||||
assert_eq!(plain(PathBuf::from(r"\\?\C:\a")), PathBuf::from(r"\\?\C:\a"), "other systems' paths stay as they are");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,666 @@
|
||||
//! The walker threads of a folder search and what they share.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
|
||||
use std::sync::{Arc, Condvar, Mutex, MutexGuard, OnceLock};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use super::rules::{Refusal, Rules, Siblings, plain};
|
||||
|
||||
/// The most walker threads one search starts.
|
||||
const MAX_THREADS: usize = 8;
|
||||
|
||||
/// The most walker threads all searches together have running.
|
||||
const MAX_LIVE: usize = 32;
|
||||
|
||||
/// How often a walker with nothing to do looks again (the deadline, the end).
|
||||
const IDLE: Duration = Duration::from_millis(50);
|
||||
|
||||
/// How many entries of a folder a walker lists between looks at the end, the deadline and the
|
||||
/// folders waiting to be listed.
|
||||
const CHECK_EVERY: u64 = 1024;
|
||||
|
||||
/// How a search ends when every walker thread stopped on a bug.
|
||||
const INTERNAL: &str = "every search thread stopped on an internal error (please report this bug)";
|
||||
|
||||
/// What a search looks for.
|
||||
pub trait Visitor: Send + Sync {
|
||||
/// Whether the regular file named `name` is to be read: asked of the files a folder lists until
|
||||
/// it has one more of them to read than [`Limits::files`].
|
||||
fn wants(&self, name: &str) -> bool;
|
||||
/// Whether the file at `path`, which [`Self::wants`] took by its name, is to be read after all,
|
||||
/// by what the file itself is. A file turned down here doesn't count among the files read.
|
||||
fn worth_reading(&self, _path: &Path) -> bool {
|
||||
true
|
||||
}
|
||||
/// The wanted items (indexes below the search's item count) the file at `path` is.
|
||||
fn items(&self, path: &Path) -> Vec<usize>;
|
||||
}
|
||||
|
||||
/// The caps on a search.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub struct Limits {
|
||||
/// Seconds before it ends (0 to 3600; 60 when not finite).
|
||||
pub seconds: f64,
|
||||
/// Entries listed, in every folder.
|
||||
pub entries: u64,
|
||||
/// Folders waiting to be listed.
|
||||
pub folders: usize,
|
||||
/// Folders below the picked one.
|
||||
pub depth: u32,
|
||||
/// Files read.
|
||||
pub files: u64,
|
||||
/// Files kept for each item.
|
||||
pub per_item: usize,
|
||||
/// Files kept in all.
|
||||
pub hits: usize,
|
||||
}
|
||||
|
||||
impl Default for Limits {
|
||||
fn default() -> Self {
|
||||
Self { seconds: 60.0, entries: 10_000_000, folders: 1_000_000, depth: 64, files: 10_000, per_item: 10, hits: 1_000 }
|
||||
}
|
||||
}
|
||||
|
||||
/// How a search ended.
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub enum End {
|
||||
/// Every folder it could enter was searched.
|
||||
Done,
|
||||
/// Every item has a file.
|
||||
Found,
|
||||
/// [`Search::stop`].
|
||||
Stopped,
|
||||
/// [`Limits::seconds`] passed.
|
||||
Time,
|
||||
/// The entries listed, the folders waiting to be listed, the files read or the files kept
|
||||
/// reached their [`Limits`]. Folders deeper than [`Limits::depth`] are skipped and an item
|
||||
/// keeps at most [`Limits::per_item`] files, without ending the search.
|
||||
Limit,
|
||||
/// It couldn't start: the folder can't be read, isn't a folder or isn't one to search.
|
||||
Failed(String),
|
||||
}
|
||||
|
||||
/// Where a search stands ([`Search::progress`]).
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct Progress {
|
||||
/// `None` while it runs.
|
||||
pub end: Option<End>,
|
||||
/// The folder searched (canonical, through [`plain`]), once it was checked. The walk reads the
|
||||
/// folder and the files below it by the paths with the `\\?\` prefix that Windows gives them.
|
||||
pub folder: PathBuf,
|
||||
/// The files found, each with the items it is, in the order found (through [`plain`]).
|
||||
pub hits: Vec<(PathBuf, Vec<usize>)>,
|
||||
/// Folders listed.
|
||||
pub folders: u64,
|
||||
/// Files listed.
|
||||
pub files: u64,
|
||||
/// Files read.
|
||||
pub read: u64,
|
||||
/// Folders not entered (other apps', the system's, hidden ones, ones too deep, ones whose
|
||||
/// contents aren't on the disk, ones whose names can't be read).
|
||||
pub skipped: u64,
|
||||
/// Folders and files that couldn't be read.
|
||||
pub unreadable: u64,
|
||||
/// How long it ran, or has been running.
|
||||
pub seconds: f64,
|
||||
}
|
||||
|
||||
impl Progress {
|
||||
/// `searching`, `done` ([`End::Done`], [`End::Found`]), `stopped` ([`End::Stopped`],
|
||||
/// [`End::Time`], [`End::Limit`]) or `failed`.
|
||||
pub fn state(&self) -> &'static str {
|
||||
match &self.end {
|
||||
None => "searching",
|
||||
Some(End::Done | End::Found) => "done",
|
||||
Some(End::Stopped | End::Time | End::Limit) => "stopped",
|
||||
Some(End::Failed(_)) => "failed",
|
||||
}
|
||||
}
|
||||
|
||||
/// Why a stopped search stopped: `stop`, `time` or `limit`.
|
||||
pub fn stopped(&self) -> Option<&'static str> {
|
||||
match &self.end {
|
||||
Some(End::Stopped) => Some("stop"),
|
||||
Some(End::Time) => Some("time"),
|
||||
Some(End::Limit) => Some("limit"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Why a failed search failed.
|
||||
pub fn error(&self) -> Option<&str> {
|
||||
match &self.end {
|
||||
Some(End::Failed(e)) => Some(e),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Walker threads shared out among searches: all of them together run at most `cap` threads.
|
||||
pub(crate) struct Pool {
|
||||
live: AtomicUsize,
|
||||
cap: usize,
|
||||
}
|
||||
|
||||
impl Pool {
|
||||
pub(crate) const fn new(cap: usize) -> Self {
|
||||
Self { live: AtomicUsize::new(0), cap }
|
||||
}
|
||||
|
||||
/// Reserve up to `want` threads: as many as are free, none when none is.
|
||||
fn reserve(&self, want: usize) -> usize {
|
||||
let mut live = self.live.load(Ordering::SeqCst);
|
||||
loop {
|
||||
let n = want.min(self.cap.saturating_sub(live));
|
||||
if n == 0 {
|
||||
return 0;
|
||||
}
|
||||
match self.live.compare_exchange(live, live + n, Ordering::SeqCst, Ordering::SeqCst) {
|
||||
Ok(_) => return n,
|
||||
Err(now) => live = now,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn release(&self, n: usize) {
|
||||
self.live.fetch_sub(n, Ordering::SeqCst);
|
||||
}
|
||||
|
||||
/// The threads running now.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn live(&self) -> usize {
|
||||
self.live.load(Ordering::SeqCst)
|
||||
}
|
||||
}
|
||||
|
||||
/// The threads of every search.
|
||||
static POOL: Pool = Pool::new(MAX_LIVE);
|
||||
|
||||
/// The walker threads for a search: the logical CPUs less one and less the renderer's threads,
|
||||
/// from 2 to 8.
|
||||
pub fn threads() -> usize {
|
||||
let cpus = std::thread::available_parallelism().map_or(2, std::num::NonZero::get);
|
||||
threads_for(cpus, usize::from(vectorcraft_render::default_threads()))
|
||||
}
|
||||
|
||||
/// [`threads`] on `cpus` logical CPUs with `render` renderer threads.
|
||||
pub(crate) fn threads_for(cpus: usize, render: usize) -> usize {
|
||||
cpus.saturating_sub(1 + render).clamp(2, MAX_THREADS)
|
||||
}
|
||||
|
||||
/// A running (or ended) search. Dropping it stops it.
|
||||
pub struct Search {
|
||||
shared: Arc<Shared>,
|
||||
}
|
||||
|
||||
impl Search {
|
||||
/// End the search ([`End::Stopped`]) unless it has ended. Walkers busy reading a folder or a
|
||||
/// file stop after it.
|
||||
pub fn stop(&self) {
|
||||
let mut w = self.shared.lock();
|
||||
self.shared.finish(&mut w, End::Stopped);
|
||||
}
|
||||
|
||||
/// Where the search stands. Past its deadline it has ended ([`End::Time`]), even while a walker
|
||||
/// is still waiting for a slow folder or file.
|
||||
pub fn progress(&self) -> Progress {
|
||||
let s = &self.shared;
|
||||
let mut w = s.lock();
|
||||
if w.end.is_none() && Instant::now() >= s.deadline {
|
||||
s.finish(&mut w, End::Time);
|
||||
}
|
||||
Progress {
|
||||
end: w.end.clone(),
|
||||
folder: plain(w.folder.clone()),
|
||||
hits: w.hits.iter().map(|(path, items)| (plain(path.clone()), items.clone())).collect(),
|
||||
folders: w.folders,
|
||||
files: w.files,
|
||||
read: s.read.load(Ordering::Relaxed).min(s.limits.files),
|
||||
skipped: w.skipped,
|
||||
unreadable: w.unreadable,
|
||||
seconds: if w.end.is_some() { w.seconds } else { s.started.elapsed().as_secs_f64() },
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Search {
|
||||
fn drop(&mut self) {
|
||||
self.stop();
|
||||
}
|
||||
}
|
||||
|
||||
/// Search `folder` and its subfolders on up to `threads` walker threads (at most 8, as many as the
|
||||
/// process-wide cap leaves) for the `items` a `visitor` looks for, where `rules` allow (this
|
||||
/// computer's when `None`). Returns at once; the folder is checked on a walker thread.
|
||||
pub fn start(
|
||||
folder: PathBuf,
|
||||
rules: Option<Rules>,
|
||||
limits: Limits,
|
||||
threads: usize,
|
||||
items: usize,
|
||||
visitor: Arc<dyn Visitor>,
|
||||
) -> Result<Search, String> {
|
||||
start_in(&POOL, folder, rules, limits, threads, items, visitor)
|
||||
}
|
||||
|
||||
/// [`start`] with the threads of `pool`.
|
||||
pub(crate) fn start_in(
|
||||
pool: &'static Pool,
|
||||
folder: PathBuf,
|
||||
rules: Option<Rules>,
|
||||
limits: Limits,
|
||||
threads: usize,
|
||||
items: usize,
|
||||
visitor: Arc<dyn Visitor>,
|
||||
) -> Result<Search, String> {
|
||||
if cfg!(target_arch = "wasm32") {
|
||||
return Err("searching a folder isn't available on the web".into());
|
||||
}
|
||||
let seconds = if limits.seconds.is_finite() { limits.seconds.clamp(0.0, 3600.0) } else { 60.0 };
|
||||
let n = pool.reserve(threads.clamp(1, MAX_THREADS));
|
||||
if n == 0 {
|
||||
return Err("the search couldn't start: try again in a moment".into());
|
||||
}
|
||||
let started = Instant::now();
|
||||
let shared = Arc::new(Shared {
|
||||
stop: AtomicBool::new(false),
|
||||
entries: AtomicU64::new(0),
|
||||
read: AtomicU64::new(0),
|
||||
started,
|
||||
deadline: started + Duration::from_secs_f64(seconds),
|
||||
limits,
|
||||
items,
|
||||
given: rules,
|
||||
rules: OnceLock::new(),
|
||||
visitor,
|
||||
walk: Mutex::new(Walk { root: Some(folder), alive: n, found: vec![0; items], ..Walk::default() }),
|
||||
wake: Condvar::new(),
|
||||
pool,
|
||||
});
|
||||
let mut spawned = 0;
|
||||
for _ in 0..n {
|
||||
let s = shared.clone();
|
||||
let thread = std::thread::Builder::new().name("vectorcraft-find".into()).spawn(move || {
|
||||
let _alive = Alive(s.clone());
|
||||
// A bug in the walk ends this walker and the others go on; when every walker ends this
|
||||
// way, the search fails ([`Alive`]).
|
||||
let _ = crate::guard::catch_panic(|| walker(&s));
|
||||
});
|
||||
if thread.is_ok() {
|
||||
spawned += 1;
|
||||
}
|
||||
}
|
||||
if spawned < n {
|
||||
pool.release(n - spawned);
|
||||
let mut w = shared.lock();
|
||||
w.alive = w.alive.saturating_sub(n - spawned);
|
||||
if spawned == 0 {
|
||||
return Err("the search couldn't start: try again in a moment".into());
|
||||
}
|
||||
if w.alive == 0 && w.end.is_none() {
|
||||
shared.finish(&mut w, End::Failed(INTERNAL.into()));
|
||||
}
|
||||
}
|
||||
Ok(Search { shared })
|
||||
}
|
||||
|
||||
/// What a search's walkers share.
|
||||
struct Shared {
|
||||
/// Set once the search ended: walkers stop listing and reading.
|
||||
stop: AtomicBool,
|
||||
entries: AtomicU64,
|
||||
read: AtomicU64,
|
||||
started: Instant,
|
||||
deadline: Instant,
|
||||
limits: Limits,
|
||||
items: usize,
|
||||
/// The rules given to [`start`], else this computer's, read by the first walker.
|
||||
given: Option<Rules>,
|
||||
rules: OnceLock<Rules>,
|
||||
visitor: Arc<dyn Visitor>,
|
||||
walk: Mutex<Walk>,
|
||||
/// Woken when folders are queued, a walker finishes a folder, or the search ends.
|
||||
wake: Condvar,
|
||||
pool: &'static Pool,
|
||||
}
|
||||
|
||||
/// The state walkers change under the lock.
|
||||
#[derive(Default)]
|
||||
struct Walk {
|
||||
/// The picked folder, until a walker takes it to check it.
|
||||
root: Option<PathBuf>,
|
||||
/// A walker is checking the picked folder.
|
||||
preparing: bool,
|
||||
folder: PathBuf,
|
||||
/// Folders waiting to be listed, with their depth below the picked one.
|
||||
stack: Vec<(PathBuf, u32)>,
|
||||
/// Folders being listed (and their candidates read) now.
|
||||
listing: usize,
|
||||
/// Walker threads still running.
|
||||
alive: usize,
|
||||
end: Option<End>,
|
||||
seconds: f64,
|
||||
hits: Vec<(PathBuf, Vec<usize>)>,
|
||||
/// Files kept for each item.
|
||||
found: Vec<usize>,
|
||||
folders: u64,
|
||||
files: u64,
|
||||
skipped: u64,
|
||||
unreadable: u64,
|
||||
/// The home folder was queued from a system's root ([`Rules::home_below`]).
|
||||
home_queued: bool,
|
||||
}
|
||||
|
||||
impl Shared {
|
||||
fn lock(&self) -> MutexGuard<'_, Walk> {
|
||||
self.walk.lock().unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
|
||||
/// End the search as `end` unless it has ended, and wake every walker to see it.
|
||||
fn finish(&self, w: &mut Walk, end: End) {
|
||||
if w.end.is_none() {
|
||||
w.end = Some(end);
|
||||
w.seconds = self.started.elapsed().as_secs_f64();
|
||||
}
|
||||
self.stop.store(true, Ordering::Relaxed);
|
||||
self.wake.notify_all();
|
||||
}
|
||||
|
||||
fn end_with(&self, end: End) {
|
||||
let mut w = self.lock();
|
||||
self.finish(&mut w, end);
|
||||
}
|
||||
|
||||
/// Whether to stop listing and reading: the search ended or its time is up.
|
||||
fn should_stop(&self) -> bool {
|
||||
self.stop.load(Ordering::Relaxed) || Instant::now() >= self.deadline
|
||||
}
|
||||
|
||||
/// Whether the folders waiting to be listed and `more` folders found in the folder being listed
|
||||
/// come to more than [`Limits::folders`], which ends the search ([`End::Limit`]).
|
||||
fn too_many_folders(&self, more: usize) -> bool {
|
||||
let mut w = self.lock();
|
||||
let over = w.stack.len().saturating_add(more) > self.limits.folders;
|
||||
if over {
|
||||
self.finish(&mut w, End::Limit);
|
||||
}
|
||||
over
|
||||
}
|
||||
|
||||
/// Check the picked folder, then queue it.
|
||||
fn prepare(&self, root: &Path) {
|
||||
let rules = self.rules.get_or_init(|| self.given.clone().unwrap_or_else(Rules::system));
|
||||
let path = root.display();
|
||||
let checked = match std::fs::canonicalize(root) {
|
||||
Err(e) => Err(format!("{path}: {e}")),
|
||||
Ok(folder) if !folder.is_dir() => Err(format!("{path} is not a folder")),
|
||||
Ok(folder) => match rules.refusal(&folder) {
|
||||
None => Ok(folder),
|
||||
Some(Refusal::Shared) => Err(format!("VectorCraft doesn't search the Shared or Public folder in Users: {path}")),
|
||||
Some(Refusal::Apps) => Err(format!("VectorCraft doesn't search other apps' or the system's folders: {path}")),
|
||||
},
|
||||
};
|
||||
let mut w = self.lock();
|
||||
match checked {
|
||||
Ok(folder) => {
|
||||
w.folder = folder.clone();
|
||||
w.stack.push((folder, 0));
|
||||
}
|
||||
Err(e) => self.finish(&mut w, End::Failed(e)),
|
||||
}
|
||||
}
|
||||
|
||||
/// List one folder: queue its subfolders, then read the files [`Visitor::wants`] accepts.
|
||||
fn list(&self, dir: &Path, depth: u32) {
|
||||
let Some(rules) = self.rules.get() else { return };
|
||||
let (mut skipped, mut unreadable, mut files) = (0, 0, 0);
|
||||
let entries = match std::fs::read_dir(dir) {
|
||||
Ok(entries) => entries,
|
||||
Err(e) => {
|
||||
let mut w = self.lock();
|
||||
// The picked folder itself: the search can't run, and says why.
|
||||
if depth == 0 {
|
||||
self.finish(&mut w, End::Failed(format!("{}: {e}", plain(dir.to_path_buf()).display())));
|
||||
} else {
|
||||
w.unreadable += 1;
|
||||
w.folders += 1;
|
||||
}
|
||||
return;
|
||||
}
|
||||
};
|
||||
let (mut folders, mut siblings, mut candidates) = (vec![], Siblings::default(), vec![]);
|
||||
for (n, entry) in entries.enumerate() {
|
||||
if n as u64 % CHECK_EVERY == CHECK_EVERY - 1 && (self.should_stop() || self.too_many_folders(folders.len())) {
|
||||
break;
|
||||
}
|
||||
if self.entries.fetch_add(1, Ordering::Relaxed) >= self.limits.entries {
|
||||
self.end_with(End::Limit);
|
||||
break;
|
||||
}
|
||||
let Ok(entry) = entry else {
|
||||
unreadable += 1;
|
||||
break;
|
||||
};
|
||||
let Ok(kind) = entry.file_type() else {
|
||||
unreadable += 1;
|
||||
continue;
|
||||
};
|
||||
let name = entry.file_name();
|
||||
let Some(name) = name.to_str() else {
|
||||
// A name that isn't Unicode: its folder is left out, its file can't be wanted.
|
||||
if kind.is_dir() {
|
||||
skipped += 1;
|
||||
}
|
||||
continue;
|
||||
};
|
||||
if kind.is_dir() || kind.is_symlink() {
|
||||
siblings.note(name);
|
||||
}
|
||||
// Links are neither followed nor read.
|
||||
if kind.is_dir() {
|
||||
folders.push((entry, name.to_string()));
|
||||
} else if kind.is_file() {
|
||||
files += 1;
|
||||
// At most one file more than the search reads is kept to be read.
|
||||
if (candidates.len() as u64) <= self.limits.files && self.visitor.wants(name) && self.visitor.worth_reading(&entry.path()) {
|
||||
if hidden_and_offline(&entry).1 {
|
||||
// Its contents are in the cloud: counted as unreadable, never read.
|
||||
unreadable += 1;
|
||||
} else {
|
||||
candidates.push(entry);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// The folder is told once for all its folders.
|
||||
let listed = rules.listed(dir, siblings);
|
||||
let mut queue = vec![];
|
||||
for (n, (entry, name)) in folders.into_iter().enumerate() {
|
||||
if n as u64 % CHECK_EVERY == CHECK_EVERY - 1 && self.should_stop() {
|
||||
break;
|
||||
}
|
||||
let path = entry.path();
|
||||
let (hidden, offline) = hidden_and_offline(&entry);
|
||||
if depth >= self.limits.depth || name.starts_with('.') || hidden || offline || rules.never_in(&listed, &name) || rules.is_refused(&path) {
|
||||
skipped += 1;
|
||||
continue;
|
||||
}
|
||||
queue.push((path, depth + 1));
|
||||
}
|
||||
let home = rules.home_below(&listed);
|
||||
{
|
||||
let mut w = self.lock();
|
||||
w.files += files;
|
||||
w.skipped += skipped;
|
||||
w.unreadable += unreadable;
|
||||
if w.end.is_none() {
|
||||
if w.stack.len() + queue.len() > self.limits.folders {
|
||||
self.finish(&mut w, End::Limit);
|
||||
} else {
|
||||
w.stack.extend(queue);
|
||||
if let Some(home) = home.filter(|_| !w.home_queued) {
|
||||
w.home_queued = true;
|
||||
w.stack.push((home, 1));
|
||||
}
|
||||
self.wake.notify_all();
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut unreadable = 0;
|
||||
for entry in candidates {
|
||||
if self.should_stop() {
|
||||
break;
|
||||
}
|
||||
if self.read.fetch_add(1, Ordering::Relaxed) >= self.limits.files {
|
||||
self.end_with(End::Limit);
|
||||
break;
|
||||
}
|
||||
let path = entry.path();
|
||||
match crate::guard::catch_panic(|| self.visitor.items(&path)) {
|
||||
Ok(items) if !items.is_empty() => self.keep(path, items),
|
||||
Ok(_) => {}
|
||||
// A file that trips a bug: as one that can't be read, and the search goes on.
|
||||
Err(_) => unreadable += 1,
|
||||
}
|
||||
}
|
||||
let mut w = self.lock();
|
||||
w.folders += 1;
|
||||
w.unreadable += unreadable;
|
||||
}
|
||||
|
||||
/// Keep a file the visitor gave `items` for, for those of its items that take more files, as
|
||||
/// soon as it is read: a file read as the search ends is kept too. The search ends at the most
|
||||
/// files it keeps, and once every item has a file.
|
||||
fn keep(&self, path: PathBuf, mut items: Vec<usize>) {
|
||||
let mut w = self.lock();
|
||||
items.sort_unstable();
|
||||
items.dedup();
|
||||
items.retain(|i| w.found.get(*i).is_some_and(|n| *n < self.limits.per_item));
|
||||
if items.is_empty() || w.hits.len() >= self.limits.hits {
|
||||
return;
|
||||
}
|
||||
for i in &items {
|
||||
if let Some(n) = w.found.get_mut(*i) {
|
||||
*n += 1;
|
||||
}
|
||||
}
|
||||
w.hits.push((path, items));
|
||||
if w.hits.len() >= self.limits.hits {
|
||||
self.finish(&mut w, End::Limit);
|
||||
}
|
||||
if self.items > 0 && w.found.iter().all(|n| *n > 0) {
|
||||
self.finish(&mut w, End::Found);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What a walker does next.
|
||||
enum Job {
|
||||
Prepare(PathBuf),
|
||||
List(PathBuf, u32),
|
||||
}
|
||||
|
||||
/// A walker thread: check the picked folder, or list the next folder, until the search ends.
|
||||
fn walker(s: &Shared) {
|
||||
loop {
|
||||
let job = {
|
||||
let mut w = s.lock();
|
||||
loop {
|
||||
if w.end.is_some() {
|
||||
return;
|
||||
}
|
||||
if Instant::now() >= s.deadline {
|
||||
s.finish(&mut w, End::Time);
|
||||
return;
|
||||
}
|
||||
if let Some(root) = w.root.take() {
|
||||
w.preparing = true;
|
||||
break Job::Prepare(root);
|
||||
}
|
||||
if let Some((dir, depth)) = w.stack.pop() {
|
||||
w.listing += 1;
|
||||
break Job::List(dir, depth);
|
||||
}
|
||||
if !w.preparing && w.listing == 0 {
|
||||
s.finish(&mut w, End::Done);
|
||||
return;
|
||||
}
|
||||
w = s.wake.wait_timeout(w, IDLE).map_or_else(|e| e.into_inner().0, |(w, _)| w);
|
||||
}
|
||||
};
|
||||
// The guard lowers `preparing` or `listing` when the job ends, on a bug too.
|
||||
let _busy = Busy(s, matches!(job, Job::Prepare(_)));
|
||||
let done = crate::guard::catch_panic(|| match &job {
|
||||
Job::Prepare(root) => s.prepare(root),
|
||||
Job::List(dir, depth) => s.list(dir, *depth),
|
||||
});
|
||||
if done.is_err() {
|
||||
let mut w = s.lock();
|
||||
match job {
|
||||
Job::Prepare(_) => s.finish(&mut w, End::Failed(INTERNAL.into())),
|
||||
Job::List(..) => w.unreadable += 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A walker's job in progress: preparing (`true`) or listing a folder.
|
||||
struct Busy<'a>(&'a Shared, bool);
|
||||
|
||||
impl Drop for Busy<'_> {
|
||||
fn drop(&mut self) {
|
||||
let mut w = self.0.lock();
|
||||
if self.1 {
|
||||
w.preparing = false;
|
||||
} else {
|
||||
w.listing = w.listing.saturating_sub(1);
|
||||
}
|
||||
self.0.wake.notify_all();
|
||||
}
|
||||
}
|
||||
|
||||
/// A running walker thread, counted out of the search and the pool when it ends.
|
||||
struct Alive(Arc<Shared>);
|
||||
|
||||
impl Drop for Alive {
|
||||
fn drop(&mut self) {
|
||||
let mut w = self.0.lock();
|
||||
w.alive = w.alive.saturating_sub(1);
|
||||
if w.alive == 0 && w.end.is_none() {
|
||||
self.0.finish(&mut w, End::Failed(INTERNAL.into()));
|
||||
}
|
||||
drop(w);
|
||||
self.0.pool.release(1);
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a folder or file is hidden by its flags (macOS `UF_HIDDEN`; Windows hidden or system),
|
||||
/// and whether its contents are in the cloud rather than on the disk (macOS `SF_DATALESS`; Windows
|
||||
/// offline or recalled on access).
|
||||
#[cfg(target_os = "macos")]
|
||||
fn hidden_and_offline(entry: &std::fs::DirEntry) -> (bool, bool) {
|
||||
use std::os::macos::fs::MetadataExt;
|
||||
const UF_HIDDEN: u32 = 0x8000;
|
||||
const SF_DATALESS: u32 = 0x4000_0000;
|
||||
entry.metadata().map_or((false, false), |m| (m.st_flags() & UF_HIDDEN != 0, m.st_flags() & SF_DATALESS != 0))
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn hidden_and_offline(entry: &std::fs::DirEntry) -> (bool, bool) {
|
||||
use std::os::windows::fs::MetadataExt;
|
||||
const HIDDEN: u32 = 0x2;
|
||||
const SYSTEM: u32 = 0x4;
|
||||
const OFFLINE: u32 = 0x1000;
|
||||
const RECALL_ON_OPEN: u32 = 0x4_0000;
|
||||
const RECALL_ON_DATA_ACCESS: u32 = 0x40_0000;
|
||||
entry.metadata().map_or((false, false), |m| {
|
||||
let a = m.file_attributes();
|
||||
(a & (HIDDEN | SYSTEM) != 0, a & (OFFLINE | RECALL_ON_OPEN | RECALL_ON_DATA_ACCESS) != 0)
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(not(any(target_os = "macos", windows)))]
|
||||
fn hidden_and_offline(_: &std::fs::DirEntry) -> (bool, bool) {
|
||||
(false, false)
|
||||
}
|
||||
@@ -0,0 +1,515 @@
|
||||
//! The fonts a document uses that aren't available (`text.missingFonts`), the files that provide
|
||||
//! them in a folder the user picks (`text.findFontFiles`, searched on separate threads:
|
||||
//! [`super::findfiles`]), and copying the files the user chooses into VectorCraft's own Fonts
|
||||
//! folder, which the font scan reads (`text.addFontFiles`).
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use serde_json::{Value, json};
|
||||
use vectorcraft_doc::Document;
|
||||
use vectorcraft_text::{FontDb, FontMatch, WantedFont, WantedFonts};
|
||||
|
||||
use super::findfiles::{self, Limits, Visitor};
|
||||
use super::*;
|
||||
use crate::DocState;
|
||||
|
||||
/// The most fonts listed, or looked for at once.
|
||||
const MAX_FONTS: usize = 1_000;
|
||||
/// The longest family or style name listed or given (bytes).
|
||||
const MAX_NAME: usize = 256;
|
||||
/// The CSS generic font families (CSS Fonts Module Level 4), which SVG and HTML files name in place
|
||||
/// of a font and no font file provides; compared in any ASCII case. The list leaves out
|
||||
/// `fangsong`, which is also the family name of a Chinese font (FangSong).
|
||||
const GENERIC_FAMILIES: &[&str] = &[
|
||||
"serif",
|
||||
"sans-serif",
|
||||
"monospace",
|
||||
"cursive",
|
||||
"fantasy",
|
||||
"system-ui",
|
||||
"ui-serif",
|
||||
"ui-sans-serif",
|
||||
"ui-monospace",
|
||||
"ui-rounded",
|
||||
"math",
|
||||
"emoji",
|
||||
];
|
||||
/// The longest folder path given (bytes).
|
||||
const MAX_PATH: usize = 4096;
|
||||
/// The most files one `text.addFontFiles` copies: as many as a search keeps.
|
||||
const MAX_ADD_FILES: usize = 1_000;
|
||||
/// The largest font file copied, and the most bytes one `text.addFontFiles` copies.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
pub(crate) const MAX_FONT_FILE: u64 = 256 << 20;
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
const MAX_ADD_BYTES: u64 = 1 << 30;
|
||||
/// The highest number a copy's name gets (`Name 99.otf`).
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
const MAX_NUMBER: u32 = 99;
|
||||
/// The extensions of the font files `text.addFontFiles` copies (in any case), besides suitcase
|
||||
/// fonts.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
const FONT_EXTENSIONS: &[&str] = &["ttf", "otf", "ttc", "otc", "woff", "woff2"];
|
||||
/// Why a file wasn't copied.
|
||||
const NOT_A_FILE: &str = "not a file";
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
const NOT_A_FONT: &str = "not a font file";
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
const TOO_LARGE: &str = "larger than 256 MB";
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
const TOO_MUCH: &str = "the files come to more than 1 GB: add fewer at a time";
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
const NO_NAME: &str = "every numbered name for it is taken in the Fonts folder";
|
||||
|
||||
pub fn specs() -> Vec<CommandSpec> {
|
||||
vec![
|
||||
cmd!(
|
||||
query "text.missingFonts",
|
||||
"Missing Fonts",
|
||||
[],
|
||||
None,
|
||||
"{} → {fonts: [{family, style, status: missing|substitute, resolved: {family, style}}] (sorted by name; at most 1000), count, fontsNextToDocument?} the fonts the active document's type uses, in its layers and its symbols, that aren't available as named: missing (the family is unknown and the fallback family stands in) or substitute (the family is there but not the style; resolved names the stand-in), as text.fonts says. CSS generic families such as sans-serif and names longer than 256 bytes are left out. fontsNextToDocument: the Fonts folder next to the saved document (File › Package writes it), when there is one and a search may start there (not in another app's folders, such as a mail app's attachments)",
|
||||
has_doc,
|
||||
missing
|
||||
),
|
||||
cmd!(
|
||||
query "text.findFontFiles",
|
||||
"Find Font Files",
|
||||
[],
|
||||
None,
|
||||
"{folder?: an absolute path, fonts?: [{family, style?}] (1 to 1000; default: the active document's missing fonts, as text.missingFonts lists them), maxSeconds?: 60 (1 to 600), stop?: false (true: stop the last search)} with folder: look in it and its subfolders for font files that make the fonts available, on separate threads, and answer at once (state: searching); without folder: the last search's state, with the files found so far. A search reads the table directories and the name, fvar and OS/2 tables of .ttf, .otf, .ttc and .otc files and, on macOS, of suitcase fonts (files without an extension whose fonts are in their resource fork), and lists a file for a font when the file, added with text.addFontFiles, makes the font resolve exactly as text.fonts resolves it; fonts that resolve exactly already are left out. It doesn't enter app packages, folders named Program Files (also x86 and Arm), ProgramData, $Recycle.Bin or System Volume Information, the system's folders at the root of a volume, the Shared and Public folders in Users, a home folder's Library, AppData, Applications, snap and dot folders, a Library folder's app data folders or the folders the environment names for apps' data, and a search of a folder inside one of them fails; the walk also skips other hidden folders, folders whose contents are in the cloud and folders more than 64 levels deep, and follows no links (a picked path is resolved through its links); none on the web → {id (absent while idle), state: idle|searching|done|stopped|failed, folder?, fonts: [{family, style, status, files: [paths]}], searched?: {folders, files, fontFiles}, skipped?, unreadable?, stopped?: stop|time|limit, error?, seconds?}",
|
||||
always,
|
||||
find
|
||||
),
|
||||
cmd!(
|
||||
query "text.addFontFiles",
|
||||
"Add Font Files",
|
||||
[],
|
||||
None,
|
||||
"{files: [paths the last text.findFontFiles search found] (1 to 1000)} copy the font files into VectorCraft's own Fonts folder (next to the preferences; the desktop app and vectorcraft-cli read it), never replacing a file there: a file with the same contents is kept, and when another file has that name, the copy gets a number (Name 2.otf); then scan the fonts again, as text.rescanFonts does, and type set in them redraws. Only regular .ttf, .otf, .ttc, .otc, .woff and .woff2 files and suitcase fonts (with their resource fork) are copied, at most 256 MB a file and 1 GB in all; any other file is skipped. Add only fonts you own or are licensed to install; none on the web → {folder, copied: [{from, to}], kept: [{from, to}], skipped: [{file, reason}], families?, faces?}",
|
||||
always,
|
||||
add
|
||||
),
|
||||
]
|
||||
}
|
||||
|
||||
/// A font a document names that isn't available as named.
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct MissingFont {
|
||||
pub family: String,
|
||||
pub style: String,
|
||||
/// [`FontMatch::Missing`] or [`FontMatch::Style`].
|
||||
pub status: FontMatch,
|
||||
/// The family and style of the face shown instead.
|
||||
pub resolved: Option<(String, String)>,
|
||||
}
|
||||
|
||||
/// The fonts the type of `d` uses, in its layers and its symbols, that aren't available as named
|
||||
/// (sorted by family and style, each once whatever version its type names; at most 1000). CSS generic families such as `sans-serif`
|
||||
/// ([`GENERIC_FAMILIES`]) and names longer than 256 bytes are left out.
|
||||
pub fn missing_fonts(d: &Document) -> Vec<MissingFont> {
|
||||
let db = FontDb::global();
|
||||
let generic = |family: &str| GENERIC_FAMILIES.iter().any(|g| g.eq_ignore_ascii_case(family.trim()));
|
||||
let mut used: Vec<(String, String)> = super::fonts::used_fonts(d).into_iter().map(|(family, style, _)| (family, style)).collect();
|
||||
// Sorted by family and style: the versions type names of one face are neighbors.
|
||||
used.dedup();
|
||||
used.into_iter()
|
||||
.filter(|(family, style)| !family.trim().is_empty() && !generic(family) && family.len() <= MAX_NAME && style.len() <= MAX_NAME)
|
||||
.filter_map(|(family, style)| {
|
||||
let resolved = db.resolve(&family, &style);
|
||||
let status = resolved.as_ref().map_or(FontMatch::Missing, |(_, m)| *m);
|
||||
(status != FontMatch::Exact).then(|| MissingFont {
|
||||
family,
|
||||
style,
|
||||
status,
|
||||
resolved: resolved.map(|(f, _)| (f.family.clone(), f.style.clone())),
|
||||
})
|
||||
})
|
||||
.take(MAX_FONTS)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// `text.missingFonts`'s answer for the open document `st`, where searches go by `rules` (this
|
||||
/// computer's when `None`).
|
||||
pub fn missing_fonts_of(st: &DocState, rules: Option<&findfiles::Rules>) -> Value {
|
||||
let fonts: Vec<Value> = missing_fonts(&st.doc)
|
||||
.into_iter()
|
||||
.map(|m| {
|
||||
json!({
|
||||
"family": m.family, "style": m.style, "status": m.status.as_str(),
|
||||
"resolved": m.resolved.map(|(family, style)| json!({ "family": family, "style": style })),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
let mut out = json!({ "count": fonts.len(), "fonts": fonts });
|
||||
// File › Package puts the fonts in a `Fonts` folder next to the packaged document. The folder
|
||||
// is made absolute against the working folder when the document's path is relative. It is
|
||||
// offered when a search may start there, as far as the rules tell without listing a folder.
|
||||
if let Some(dir) = st.path.as_deref().and_then(|p| Path::new(p).parent()).map(|d| d.join("Fonts")).filter(|d| d.is_dir()) {
|
||||
let dir = super::fileio::absolute_path(&dir.to_string_lossy());
|
||||
let system;
|
||||
let rules = match rules {
|
||||
Some(rules) => rules,
|
||||
None => {
|
||||
system = findfiles::Rules::system();
|
||||
&system
|
||||
}
|
||||
};
|
||||
if std::fs::canonicalize(&dir).is_ok_and(|c| rules.refusal_unlisted(&c).is_none()) {
|
||||
out["fontsNextToDocument"] = json!(dir);
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn missing(s: &mut Session, _: &Value) -> Result<Value> {
|
||||
Ok(missing_fonts_of(s.doc()?, s.search_rules.as_ref()))
|
||||
}
|
||||
|
||||
/// A font a search looks for.
|
||||
#[derive(Clone, Debug)]
|
||||
pub(crate) struct Sought {
|
||||
pub(crate) family: String,
|
||||
pub(crate) style: String,
|
||||
pub(crate) status: FontMatch,
|
||||
}
|
||||
|
||||
/// The session's last search for font files (`text.findFontFiles`).
|
||||
pub(crate) struct FontSearch {
|
||||
/// The search's number in the session; [`stop`] takes it.
|
||||
pub(crate) id: u64,
|
||||
/// The folder as given.
|
||||
pub(crate) folder: String,
|
||||
pub(crate) fonts: Vec<Sought>,
|
||||
pub(crate) search: findfiles::Search,
|
||||
}
|
||||
|
||||
/// The files a search for fonts reads: font files, by their names ([`WantedFonts`]).
|
||||
struct FontFiles(WantedFonts);
|
||||
|
||||
impl Visitor for FontFiles {
|
||||
fn wants(&self, name: &str) -> bool {
|
||||
let path = Path::new(name);
|
||||
// A suitcase font (macOS) often has no extension.
|
||||
vectorcraft_text::is_font_file(path) || cfg!(target_os = "macos") && path.extension().is_none()
|
||||
}
|
||||
fn worth_reading(&self, path: &Path) -> bool {
|
||||
// A file without an extension is read only when it is a suitcase font.
|
||||
path.extension().is_some() || vectorcraft_text::is_suitcase(path)
|
||||
}
|
||||
fn items(&self, path: &Path) -> Vec<usize> {
|
||||
self.0.provided_by(path)
|
||||
}
|
||||
}
|
||||
|
||||
/// `[{family, style?}]`: 1 to 1000 fonts with a family of 1 to 256 bytes and a style of at most 256
|
||||
/// (Regular when left out).
|
||||
fn font_list(v: &Value) -> Option<Vec<(String, String)>> {
|
||||
let list = v.as_array().filter(|a| (1..=MAX_FONTS).contains(&a.len()))?;
|
||||
list.iter()
|
||||
.map(|f| {
|
||||
let family = f.get("family")?.as_str().filter(|s| !s.trim().is_empty() && s.len() <= MAX_NAME)?;
|
||||
let style = match f.get("style") {
|
||||
None | Some(Value::Null) => "Regular",
|
||||
Some(s) => s.as_str().filter(|s| s.len() <= MAX_NAME)?,
|
||||
};
|
||||
Some((family.to_string(), style.to_string()))
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn find(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
const C: &str = "text.findFontFiles";
|
||||
static NEXT: AtomicU64 = AtomicU64::new(1);
|
||||
if bool_or(p, "stop", false) {
|
||||
if let Some(f) = &s.font_search {
|
||||
f.search.stop();
|
||||
}
|
||||
return Ok(state(s));
|
||||
}
|
||||
let Some(folder) = p.get("folder").filter(|f| !f.is_null()) else { return Ok(state(s)) };
|
||||
let given = match p.get("fonts") {
|
||||
None | Some(Value::Null) => None,
|
||||
Some(v) => Some(font_list(v).ok_or_else(|| bad(C, "each of `fonts` is {family, style?}; 1 to 1000 of them"))?),
|
||||
};
|
||||
let seconds = p.get("maxSeconds").and_then(Value::as_f64).filter(|x| x.is_finite()).unwrap_or(60.0).clamp(1.0, 600.0);
|
||||
// Only checked here: the folder is read on a walker thread.
|
||||
let folder = folder
|
||||
.as_str()
|
||||
.filter(|f| !f.is_empty() && f.len() <= MAX_PATH && !f.contains(['\0', '\u{FFFD}']) && Path::new(f).is_absolute())
|
||||
.ok_or_else(|| bad(C, "`folder` must be an absolute folder path"))?
|
||||
.to_string();
|
||||
let fonts = match given {
|
||||
Some(fonts) => fonts,
|
||||
None => missing_fonts(&s.active().ok_or_else(|| bad(C, "give `fonts`, or open a document"))?.doc)
|
||||
.into_iter()
|
||||
.map(|m| (m.family, m.style))
|
||||
.collect(),
|
||||
};
|
||||
let db = FontDb::global();
|
||||
let (mut sought, mut wanted) = (vec![], vec![]);
|
||||
for (family, style) in fonts {
|
||||
let (status, installed) = match db.resolve(&family, &style) {
|
||||
Some((_, FontMatch::Exact)) => continue,
|
||||
Some((f, FontMatch::Style)) => (FontMatch::Style, Some(f.family.clone())),
|
||||
_ => (FontMatch::Missing, None),
|
||||
};
|
||||
wanted.push(WantedFont { family: family.clone(), style: style.clone(), installed });
|
||||
sought.push(Sought { family, style, status });
|
||||
}
|
||||
if sought.is_empty() {
|
||||
return Err(bad(C, "no fonts to look for: they are all available"));
|
||||
}
|
||||
// A new search stops the last one.
|
||||
s.font_search = None;
|
||||
let threads = s.search_threads.unwrap_or_else(findfiles::threads);
|
||||
let limits = Limits { seconds, ..Limits::default() };
|
||||
let visitor = Arc::new(FontFiles(WantedFonts::new(&wanted)));
|
||||
let search =
|
||||
findfiles::start(PathBuf::from(&folder), s.search_rules.clone(), limits, threads, sought.len(), visitor).map_err(EngineError::Other)?;
|
||||
s.font_search = Some(FontSearch { id: NEXT.fetch_add(1, Ordering::Relaxed), folder, fonts: sought, search });
|
||||
Ok(state(s))
|
||||
}
|
||||
|
||||
/// Where the session's last search stands (`text.findFontFiles` without `folder`).
|
||||
fn state(s: &Session) -> Value {
|
||||
let Some(f) = &s.font_search else { return json!({ "state": "idle", "fonts": [] }) };
|
||||
let p = f.search.progress();
|
||||
let mut files: Vec<Vec<String>> = vec![vec![]; f.fonts.len()];
|
||||
for (path, items) in &p.hits {
|
||||
for i in items {
|
||||
if let Some(v) = files.get_mut(*i) {
|
||||
v.push(path.to_string_lossy().into_owned());
|
||||
}
|
||||
}
|
||||
}
|
||||
let fonts: Vec<Value> = f
|
||||
.fonts
|
||||
.iter()
|
||||
.zip(files)
|
||||
.map(|(font, mut files)| {
|
||||
files.sort();
|
||||
json!({ "family": font.family, "style": font.style, "status": font.status.as_str(), "files": files })
|
||||
})
|
||||
.collect();
|
||||
let folder = if p.folder.as_os_str().is_empty() { f.folder.clone() } else { p.folder.to_string_lossy().into_owned() };
|
||||
let mut out = json!({
|
||||
"id": f.id, "state": p.state(), "folder": folder, "fonts": fonts,
|
||||
"searched": { "folders": p.folders, "files": p.files, "fontFiles": p.read },
|
||||
"skipped": p.skipped, "unreadable": p.unreadable, "seconds": (p.seconds * 10.0).round() / 10.0,
|
||||
});
|
||||
if let Some(why) = p.stopped() {
|
||||
out["stopped"] = json!(why);
|
||||
}
|
||||
if let Some(e) = p.error() {
|
||||
out["error"] = json!(e);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Stop the session's search when it is the one numbered `id` (`text.findFontFiles`'s `id`): a
|
||||
/// newer search, started by someone else, goes on.
|
||||
pub fn stop(s: &mut Session, id: u64) {
|
||||
if let Some(f) = s.font_search.as_ref().filter(|f| f.id == id) {
|
||||
f.search.stop();
|
||||
}
|
||||
}
|
||||
|
||||
/// What copying font files into a folder did: each file copied (from, to), kept because the same
|
||||
/// file is there already (from, the file there), or skipped (file, why).
|
||||
#[derive(Debug, Default)]
|
||||
pub(crate) struct CopyReport {
|
||||
pub(crate) copied: Vec<(PathBuf, PathBuf)>,
|
||||
pub(crate) kept: Vec<(PathBuf, PathBuf)>,
|
||||
pub(crate) skipped: Vec<(PathBuf, String)>,
|
||||
}
|
||||
|
||||
fn add(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
const C: &str = "text.addFontFiles";
|
||||
let mut files: Vec<PathBuf> = p
|
||||
.get("files")
|
||||
.and_then(Value::as_array)
|
||||
.filter(|a| (1..=MAX_ADD_FILES).contains(&a.len()))
|
||||
.and_then(|a| a.iter().map(|f| f.as_str().map(PathBuf::from)).collect::<Option<Vec<_>>>())
|
||||
.ok_or_else(|| bad(C, "`files` must be 1 to 1000 paths of font files the last text.findFontFiles search found"))?;
|
||||
files.sort();
|
||||
files.dedup();
|
||||
let dir = vectorcraft_text::app_font_dir()
|
||||
.ok_or_else(|| EngineError::Other("VectorCraft's Fonts folder isn't set here: the desktop app and vectorcraft-cli set it".into()))?;
|
||||
let found: Vec<PathBuf> =
|
||||
s.font_search.as_ref().map(|f| f.search.progress().hits.into_iter().map(|(path, _)| path).collect()).unwrap_or_default();
|
||||
if let Some(f) = files.iter().find(|f| !found.contains(f)) {
|
||||
return Err(bad(C, format!("`{}` is not a font file the last text.findFontFiles search found", f.display())));
|
||||
}
|
||||
super::fileio::create_dir(&dir.to_string_lossy())?;
|
||||
let report = copy_into(&files, dir);
|
||||
let pairs = |v: &[(PathBuf, PathBuf)]| {
|
||||
v.iter().map(|(from, to)| json!({ "from": from.to_string_lossy(), "to": to.to_string_lossy() })).collect::<Vec<_>>()
|
||||
};
|
||||
let mut out = json!({
|
||||
"folder": dir.to_string_lossy(), "copied": pairs(&report.copied), "kept": pairs(&report.kept),
|
||||
"skipped": report.skipped.iter().map(|(file, why)| json!({ "file": file.to_string_lossy(), "reason": why })).collect::<Vec<_>>(),
|
||||
});
|
||||
if !report.copied.is_empty() || !report.kept.is_empty() {
|
||||
// The fonts are cataloged and type set in them redraws, as Refresh Font List does it.
|
||||
let r = super::fonts::rescan(s, &Value::Null)?;
|
||||
out["families"] = r["families"].clone();
|
||||
out["faces"] = r["faces"].clone();
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Where [`copy_one`] put a file: copied to, or kept as, the file in the folder.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
enum Placed {
|
||||
Copied(PathBuf),
|
||||
Kept(PathBuf),
|
||||
}
|
||||
|
||||
/// Copy `files` into `dir`, never replacing a file there: a file with the same contents is kept,
|
||||
/// and when another file has the name, the copy gets the first free name from `Name 2.otf` to
|
||||
/// `Name 99.otf`. At most [`MAX_FONT_FILE`] bytes a file and 1 GB in all.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
pub(crate) fn copy_into(files: &[PathBuf], dir: &Path) -> CopyReport {
|
||||
let mut report = CopyReport::default();
|
||||
let mut total = 0;
|
||||
for from in files {
|
||||
match copy_one(from, dir, &mut total) {
|
||||
Ok(Placed::Copied(to)) => report.copied.push((from.clone(), to)),
|
||||
Ok(Placed::Kept(to)) => report.kept.push((from.clone(), to)),
|
||||
Err(why) => report.skipped.push((from.clone(), why)),
|
||||
}
|
||||
}
|
||||
report
|
||||
}
|
||||
|
||||
/// The web has no file system: nothing is copied.
|
||||
#[cfg(target_arch = "wasm32")]
|
||||
pub(crate) fn copy_into(files: &[PathBuf], _: &Path) -> CopyReport {
|
||||
CopyReport { skipped: files.iter().map(|f| (f.clone(), NOT_A_FILE.to_string())).collect(), ..CopyReport::default() }
|
||||
}
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn copy_one(from: &Path, dir: &Path, total: &mut u64) -> std::result::Result<Placed, String> {
|
||||
use std::io::ErrorKind;
|
||||
let meta = std::fs::symlink_metadata(from).map_err(|e| e.to_string())?;
|
||||
if !meta.is_file() {
|
||||
return Err(NOT_A_FILE.into());
|
||||
}
|
||||
let font = from.extension().and_then(|e| e.to_str()).is_some_and(|e| FONT_EXTENSIONS.iter().any(|x| e.eq_ignore_ascii_case(x)));
|
||||
// A suitcase font's fonts are in its resource fork, which is copied with it, whatever its name.
|
||||
let fork = match (vectorcraft_text::is_suitcase(from), font) {
|
||||
(true, _) => Some(read_fork(from)?),
|
||||
(false, true) => None,
|
||||
(false, false) => return Err(NOT_A_FONT.into()),
|
||||
};
|
||||
let len = fork.as_ref().map_or(meta.len(), |f| f.len() as u64);
|
||||
if len > MAX_FONT_FILE {
|
||||
return Err(TOO_LARGE.into());
|
||||
}
|
||||
if total.saturating_add(len) > MAX_ADD_BYTES {
|
||||
return Err(TOO_MUCH.into());
|
||||
}
|
||||
let (Some(name), Some(stem)) = (from.file_name(), from.file_stem()) else { return Err(NOT_A_FILE.into()) };
|
||||
let ext = from.extension().map(|e| format!(".{}", e.to_string_lossy())).unwrap_or_default();
|
||||
for n in 1..=MAX_NUMBER {
|
||||
let to = if n == 1 { dir.join(name) } else { dir.join(format!("{} {n}{ext}", stem.to_string_lossy())) };
|
||||
match std::fs::symlink_metadata(&to) {
|
||||
Err(e) if e.kind() == ErrorKind::NotFound => match create_copy(from, &to, fork.as_deref()) {
|
||||
Ok(()) => {
|
||||
*total += len;
|
||||
return Ok(Placed::Copied(to));
|
||||
}
|
||||
// Taken meanwhile: the next name.
|
||||
Err(e) if e.kind() == ErrorKind::AlreadyExists => {}
|
||||
Err(e) => return Err(e.to_string()),
|
||||
},
|
||||
Ok(m) if m.is_file() && same_file(from, &to, m.len(), len, fork.as_deref()) => return Ok(Placed::Kept(to)),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
Err(NO_NAME.into())
|
||||
}
|
||||
|
||||
/// Create `to` as a copy of the font file `from`, or for a suitcase font, as an empty file whose
|
||||
/// resource fork is `fork`; never in place of a file.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn create_copy(from: &Path, to: &Path, fork: Option<&[u8]>) -> std::io::Result<()> {
|
||||
let Some(fork) = fork else { return vectorcraft_format::write_new_with(to, |f| copy_capped(from, f)) };
|
||||
vectorcraft_format::write_new_with(to, |_| Ok(()))?;
|
||||
std::fs::write(fork_path(to), fork).inspect_err(|_| {
|
||||
// Best effort: the empty file just made is all there is to clean up.
|
||||
let _ = std::fs::remove_file(to);
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether the file `to` (`to_len` bytes) holds what the copy of `from` (`len` bytes, or the
|
||||
/// resource fork `fork` of a suitcase font) would.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn same_file(from: &Path, to: &Path, to_len: u64, len: u64, fork: Option<&[u8]>) -> bool {
|
||||
match fork {
|
||||
None => to_len == len && same_contents(from, to),
|
||||
Some(fork) => to_len == 0 && read_fork(to).is_ok_and(|f| f == fork),
|
||||
}
|
||||
}
|
||||
|
||||
/// The resource fork of the file at `path` (where macOS keeps a suitcase font's fonts), at most
|
||||
/// [`MAX_FONT_FILE`] bytes.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn read_fork(path: &Path) -> std::result::Result<Vec<u8>, String> {
|
||||
use std::io::Read;
|
||||
let mut fork = vec![];
|
||||
std::fs::File::open(fork_path(path)).and_then(|f| f.take(MAX_FONT_FILE + 1).read_to_end(&mut fork)).map_err(|e| e.to_string())?;
|
||||
if fork.len() as u64 > MAX_FONT_FILE {
|
||||
return Err(TOO_LARGE.into());
|
||||
}
|
||||
Ok(fork)
|
||||
}
|
||||
|
||||
/// Where macOS shows the resource fork of the file at `path`.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn fork_path(path: &Path) -> PathBuf {
|
||||
path.join("..namedfork/rsrc")
|
||||
}
|
||||
|
||||
/// Copy the file `from` into `to`, failing past [`MAX_FONT_FILE`] bytes (a file that grew since
|
||||
/// it was measured).
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn copy_capped(from: &Path, to: &mut std::fs::File) -> std::io::Result<()> {
|
||||
use std::io::Read;
|
||||
let copied = std::io::copy(&mut std::fs::File::open(from)?.take(MAX_FONT_FILE + 1), to)?;
|
||||
if copied > MAX_FONT_FILE {
|
||||
return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, TOO_LARGE));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Whether the files `a` and `b` hold the same bytes (read 64 KB at a time).
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn same_contents(a: &Path, b: &Path) -> bool {
|
||||
let (Ok(mut a), Ok(mut b)) = (std::fs::File::open(a), std::fs::File::open(b)) else { return false };
|
||||
let (mut x, mut y) = (vec![0; 1 << 16], vec![0; 1 << 16]);
|
||||
loop {
|
||||
match (fill(&mut a, &mut x), fill(&mut b, &mut y)) {
|
||||
(Ok(0), Ok(0)) => return true,
|
||||
(Ok(n), Ok(m)) if n == m && x.get(..n) == y.get(..m) => {}
|
||||
_ => return false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Read from `f` until `buf` is full or the file ends: the bytes read.
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
fn fill(f: &mut std::fs::File, buf: &mut [u8]) -> std::io::Result<usize> {
|
||||
use std::io::Read;
|
||||
let mut n = 0;
|
||||
while let Some(rest) = buf.get_mut(n..).filter(|r| !r.is_empty()) {
|
||||
match f.read(rest)? {
|
||||
0 => break,
|
||||
k => n += k,
|
||||
}
|
||||
}
|
||||
Ok(n)
|
||||
}
|
||||
@@ -22,7 +22,9 @@ mod effectcmd;
|
||||
pub mod expand;
|
||||
pub(crate) mod fileinfo;
|
||||
pub mod fileio;
|
||||
pub mod findfiles;
|
||||
pub mod flatten;
|
||||
pub mod fontfiles;
|
||||
mod fonts;
|
||||
pub(crate) mod freeform;
|
||||
pub(crate) mod gradient;
|
||||
@@ -223,6 +225,7 @@ pub fn command_specs() -> &'static [CommandSpec] {
|
||||
v.extend(inline::specs());
|
||||
v.extend(textstyles::specs());
|
||||
v.extend(fonts::specs());
|
||||
v.extend(fontfiles::specs());
|
||||
v.extend(help::specs());
|
||||
v.extend(threads::specs());
|
||||
v.extend(textwrap::specs());
|
||||
|
||||
@@ -50,6 +50,17 @@ fn font_ext(bytes: &[u8]) -> &'static str {
|
||||
}
|
||||
}
|
||||
|
||||
/// The name a font's file gets in the package's `Fonts` folder: the name of the file at `path`,
|
||||
/// with the extension the font's `data` calls for ([`font_ext`]) when that name has no font
|
||||
/// extension; without a file, `family-style` with that extension.
|
||||
fn packaged_name(path: Option<&Path>, family: &str, style: &str, data: &[u8]) -> String {
|
||||
match path.and_then(Path::file_name).map(|n| n.to_string_lossy().into_owned()) {
|
||||
Some(name) if vectorcraft_text::is_font_file(Path::new(&name)) => name,
|
||||
Some(name) => format!("{name}.{}", font_ext(data)),
|
||||
None => format!("{family}-{style}.{}", font_ext(data)).replace(|c: char| !(c.is_alphanumeric() || "-_.".contains(c)), ""),
|
||||
}
|
||||
}
|
||||
|
||||
/// The options of a package.
|
||||
struct Options {
|
||||
copy_links: bool,
|
||||
@@ -123,10 +134,7 @@ fn package(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let reason = match resolved {
|
||||
Some((f, _)) if f.embeddable() => {
|
||||
let data = f.file_data();
|
||||
let file = f.path().and_then(Path::file_name).map_or_else(
|
||||
|| format!("{}-{}.{}", f.family, f.style, font_ext(data)).replace(|c: char| !(c.is_alphanumeric() || "-_.".contains(c)), ""),
|
||||
|n| n.to_string_lossy().into_owned(),
|
||||
);
|
||||
let file = packaged_name(f.path(), &f.family, &f.style, data);
|
||||
// A collection, or a style shown in another face's file, is copied once.
|
||||
if files.insert(file.to_lowercase()) {
|
||||
entries.push((format!("Fonts/{file}"), data.to_vec()));
|
||||
@@ -258,3 +266,21 @@ pub(crate) fn zip(files: &[(String, &[u8])]) -> Result<Vec<u8>> {
|
||||
out.extend_from_slice(&0u16.to_le_bytes());
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A font read from a file without a font extension (one a font manager keeps, for example) is
|
||||
/// packaged with the extension its data calls for.
|
||||
#[test]
|
||||
fn packaged_fonts_have_a_font_extension() {
|
||||
let (otf, ttc, ttf) = (b"OTTO....".as_slice(), b"ttcf....".as_slice(), b"\0\x01\0\0....".as_slice());
|
||||
let name = |path: Option<&str>, data: &[u8]| packaged_name(path.map(Path::new), "Some Family", "Bold", data);
|
||||
assert_eq!(name(Some("/fonts/A8F3C2"), otf), "A8F3C2.otf");
|
||||
assert_eq!(name(Some("/fonts/Shared Fonts.dat"), ttc), "Shared Fonts.dat.ttc");
|
||||
assert_eq!(name(Some("/fonts/Kept.TTF"), otf), "Kept.TTF");
|
||||
assert_eq!(name(Some("/fonts/Kept.otc"), ttc), "Kept.otc");
|
||||
assert_eq!(name(None, ttf), "SomeFamily-Bold.ttf");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user